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SUMMARY 

PROBLEM 

The authoring process and authoring aids which facilitate development of 
instructional materials has recently emerged as an area of concern in the field 
of Instructional Systems Development (ISD). This is due to a growing re.ilization 
that instructional materials development is a major cost factor in developing 
instructional systems. The purpose of this study was to examine the ISD concept 
as it relates to the authoring process, and to survey, identify, and analyze 
authoring tools and procedures which aid the ISD process. 

APPROACH 

Both American and Canadian sources from 1965 to the present were searched in 
the following areas: (a) developmental studies of on- and off-line instructional 
systems; (b) evaluation and comparison of developmental methodologies for instruc- 
tional materials; (c) design and installation of instructional materials; (d) "user 
studies" involving effectiveness of a medium and comparative effectiveness of alter- 
native media; (e) information processing behavior of students; (f) analyses of 
training tasks and job requirements; and (g) revision procedures for instructional 
materials and systems, both on- and off-line. 

Computerized searches were made of data banks in the Educational Resources 
Information Center (ERIC), the National Technical Information Service (NTIS), and 
the libraries of McDonnell Douglas Corporation. Manual searches were made of 
indexes, abstracting services, and bibliographies at several local universities. 
A Selective Dissemination of Information (SDI) service . used along with personal 
communication with over seventy individuals and associations prominent in the field 
of instructional technology. 

In total, over five thousand reports were read in full-text or abstract form, 
and approximately fourteen hundred practical , development-oriented documents were' 
selected for inclusion in a computer-based information storage and retrieval system 
developed for the study. The documents were categorized by: (a) author; (b) insti- 
tution; (c) title; (d) publication date; (e) applicable ISD step; (f) type of media; 
(g) type of application; (h) type of evaluation; and (i) type of environment. 
' ' V , 
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RESULTS 

Findings of the study include: (a) more applied but unevaluated documents are 
associated with the civilian environment; (b) relatively equal numbers of applied 
and evaluated documents exist in both civilian and military environments; (c) a 
greater total number of authoring aids (tools, procedures, and methodologies) tends 
to be produced by the civilian environment; and (d) more procedures are available 
than either tools or methodologies. Six significant aspects of the authoring 
process were also identified. 

In addition, an extensive annotated bibliography of approximately sixty 
authoring tools and procedures was produced. Based on the Inter-service Procedures 
for Instructional Systems Development (IPISD) Model, the annotated bibliography 
groups documents by ISD phase and step, and provides citations, abstracts, and 
analytic information. 

CONCLUSIONS ' 

Recommendations are made for creating tools for ISD steps which have none, for 
refining procedures into tools for ISD steps where tools are more useful , end for 
developing methodologies for ISD steps where the state-of-the-art is in advance of 
current tools and procedures. 
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1.0 INTRODUCTION 

1.1 BACKGROUND . 



; Instruction al System Development (ISD) - Instructiortal Systems Develop- 

ment is an adaptation of the systems approach, a sequence of steps by which complex 
systems are broken down into smaller, more manageable problem areas that can then 
be analyzed for use in analysis, design, development, implementation, and control 
of an instructional system. The systems approach has been adapted to instruction 
in order to define such things as job activities, testing, sequencing instruction, 
and evaluating instructional systems in terms of both learners and organizational 
as well as physical components. 

The Interservice Procedures^or Instructional Systems Development (IPISD) 
provided the ISD model used in this study (TRADOC, 1975). The IPISD model consists 
of five phases and nineteen steps. Each step has both an input and output. The 
phases and steps in the IPISD model are not necessarily done on a once-and-once- 
only basis but rather provide for a continuing, iterative process of system 
development. 

1-1 -2 Authoring Tools and Procedures - Authoring tools and procedures were 
identified primarily to supply information for decision-making purposes, not to 
assess their merit. Since the authoring process is an ISD activity, the terms 
"authoring tools" and "authoring procedures" must be defined in relation to an 
ISD methodology. An ISD methodology is a general systems approach with multiple 
components which, given a certain set of constraints, is used to produce an instruc- 
tional system. An authoring procedure is a step or series of steps which operation- 
ally defines one component of the methodology. An authoring tool is an evaluated, 
self-contained product in a suitable format which is applied to the analysis, 
design, development, implementation, or control of one or more steps of a procedure. 

The autrtonrtg process is conceived as a series of separate, though related, 
activities: 

° Informatio n Gathering - the acquisition of supportive information from 
task analysis, existing training materials, instructors, and incoming 
students 

1-1 10 
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0 Information Conversion - the development of that information into properly 
sequenced packages with format and structure that enable learning to occur 
effectively and efficiently 

* 

0 Information Revision - the revision, as necessary, of the information 
packages during formative and summative evaluation based on feedback from 
students, instructors, and administrators 

0 Information Publication - the publication of those packages in a con- ; 
struction which meets the requirements of the learner and the training 
environment, e.g., a lesson intended for homework may require a different 
construction than one intended for in-class presentation. 

1.2 PURPOSE AND SCOPE 

The purpose of this literature search was to identify and analyze authoring 
tools and procedures for developing instructional materials and to identify infor- 
mation clearing houses and data banks for existing instructional materials. Both 
American and Canadian sources from 1965 to the present concerning ISD were searched 
in the following areas: 

0 Developmental studies of on- and off-line instructional systems 

0 Evaluation and comparison of development methodologies for instructional' 
materials 

0 Design and installation of instructional r.iaterials 

0 "User studies" involving effectiveness of a medium and comparative 
effectiveness of alternative media 

0 Information processing behavior of students 

0 Analyses of training tasks and job requirements 

0 Revision procedures for instructional materials and systems, both on- 
and off-line. 
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1.3 APPROACH 

Computerized searches were made of data bankj; in the Educational Resources 
Information Center (ERIC), the National Technical Information Service (NTIS), the 
Defense Documentation Center (DDC), and the libraries of the McDonnell Douglas 
Corporation. Manual searches were made of Indexes, abstracting services, and 
bibliographies at several area universities. A Selective Dissemination of Infor- 
mation (SDI) service was used along with personal communication with over seventy 
individuals and associations prominent in the field of Instructional technology. 
In total, over five thousand reports were read in full-text or abstract form, and 
approximately fourteen hundred practical, development-oriented Items were selected 
for inclusion in the computer-based information storage and retrieval system 
developed for the present study. Appendix A contains the codes whereby reports and 
other do.:uments were described and entered in the data base. 

1.4 RESULTS 

Six significant aspects of the authoring process were identified: (1} size 
and configuration of authoring teams; (2) knowledge and proficiency of course 
aut+rorsr (3) existing course structures, lesson materials, and course control 
documents; (4) type of media and type of instruction; (5) type and extent of 
evaluation and revision; and (6) text-handling and materials reproduction systems. 
Since the only subjective aspect of the authoring process is knowledge and profi- 
ciency of course authors, identification and distribution of authoring tools and 
procedures would allow course authors to develop materials from a baseline of equal 
knowledge. Thus, the other objective aspects of the authoring process could be 
categorized and manipulated in a "laboratory" (i.e., controlled) setting, and it 
should be possible to model the authoring process va**iables and to formulate 
decision rules for managing the materials development cycle with an optimum mix 
of personnel and organization. , 

: It was found that at least one authoring aid (tool, procedure, or methodology) 
was available for every ISD activity. The types of authoring aids, however, are 
unevenly distributed among ISD activities. Relatively equal numbers of authoring 
aids are produced by civilian and mil itary environments. Finally, it was found 
that more procedures than either tools or methodologies are available to analyze, 
design, develop, implement, and control the ISD process. 

12 
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An annotated bibliography of authoring aids for developing course materials 
using the .ISD approach was prepared as Appendix B. It includes an analysis of 
documentation related to ISD and the identification of information clearinghouses 
for existing instructional materials. The bibliography is in a format ready for 

, use. , ' ■ " ' 
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2.0 APPROACH 

2.1 PROBLEMS 

Although the national investment in computer-based educational training to 
date is some $2 billion, the extensive use of computer-related instructional tech- 
nology is yet to be realized. Anastasio and Morgan (1972), the Carnegie Commission 
(1972), Demerath afTd-Daniels (1973), Levien (1972), and Wilcox (cited in Hunter, 
Kastner, Rubin and Seidel, 1975) attempted to determine why computer-related tech- 
nology has not been widely adopted for instruction. Hunter et al. (1975) summarized 
their findings as follows: 

a. Materials - High-qual ity materials and materials using different instruc- 
tional techniques or strategies or addressing alternative subjects are not 
readily available. 

b. Overlooked Materials - When new instructional materials are developed, 
earlier versions of the work are automatically considered obsolete. 
Additionally, instructional materials in actual use are often overlooked. 

c- Relationship of Materials to Curricula - Lack of standardized curricula 
prevents a building-block approach to and comparative evaluation of 
different sets of instructional materials. 

d. Documentation - Clear, extensive documentation of innovative projects is 
seldom available. . , 

. e. Attitudes - Faculty members fear role reduction and dehumanization 
through use of instructional technology. Administrators feel that 
computers, media carrels, and other instructional hardware are luxuries 
which should be treated separately from the normal cost of school opera- 
tions and tlierefore, if necessary, can be dropped from the budget with 
little harm to instruction or learning. 
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f. Skilled Personnel - Technically competent special ists who can provide 
training in the uses and techniques of instructional technology are in 
short supply. A corollary is that broad formal training in instructional 
technology is not generally available. 

g- Proof of E ffectiveness - There are few examples of proven, cost-effective 
computer-based instructional systems. Coupled with the scarcity of 
evaluative measures applicable to computer-based instruction, there is 
no definitive evidence that computers improve the quality of learning. 

h. Cost - The high capital investment required to purchase or develop 

sophisticated instructional technology is a deterrent to use, especially 
in publ ic schools. 

Of these eight problems, four are concerned with instructional materials, the 
developmental use of which remains typical of the "cottage industry" approach 
criticized by Molnar: 

The large amount of uncoordinated research activities and the lacM of 
preplanned linkages between research and practice has led to the exis- 
tence of an expensive cottage industry in educational technology which 
tends to retool every academic year. Researchers and educators fre- 
quently demonstrate a strong resistance to the use of someone else's 
innovation. It has been said that if there was a Nobel prize for 
educational research, we would have to nominate an entire generation 
of researchers for their co-discovery of the wheel. (Molnar, 1971, p. 7) 

At least three factors have forced educational researchers to develop and 
apply their own unique methods to such things as job analysis, test generation, 
construction of behavioral objectives, and implementation, evaluation, and revision 

of instruction. First, many educators feel very strongly that instruction should 

have a local, indigenous quality (Demerath & Daniels, 1973). Second, instructional 
development efforts are usually driven by a "raw empiricism" so that 

Instructional materials are prepared on the basis of intuition, folklore, 
or experience and administered to members of the target population. If 
the students pass the test, the product is considered appropriate; if not, 
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the materials are revised and tried again. This tryout-revision cycle 
is repeated until the product works or the developers run out of 
resources or time. (Merrill and Boutwell, 1973, p. 96) 

Third, there is a. lack of information on available authoring tools, procedures, 
and clearinghouses for existing course materials (Logan, 1976). . 

This study attempted to identify and evaluate authoring aids and information 
clearinghouses |hat facilitate analyzing, designing, developing, implementing, and 
controlling the ISD process. In addition, four research hypotheses were tested; 

0 The civilian environment has a greater number of applied, unevaluated 
documents than the military environment 

0 The civilian environment has a greater number of applied, evaluated docu- 
ments than the military environment 

0 There are unequal numbers of documents within the. three authoring and 
categories (tools, procedures, and methodologies) 

0 The civilian environment has a greater number of authoring aids than the 
military environment. 

Authoring tools and procedures were identified primarily to supply information 
for decisior\'-making purposes, not to assess their merit. The first reason for 
taking this approach was that selection criteria did not include evaluation of the 
relative worth of a uarticular tool or procedure. Second, data resulting from 
evaluation of a particular tool or procedure were either unavailable or not clearly 
identified. And third, the study emphasized breadth, not depth, and, therefore, 
many tools and procedures in as many ISD activities as possible were identified -- 
regardless of their relative worth. 

The terms "authoring tools" and "authoring procedures" are defined in relation 
to an ISD methodology. Accordingly, three definitions are given. An ISD methodology 
is a general systems approach with multiple components which, given a certain set 
of constraints, is used to produce an instructional system. An authoring procedure 
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IS. a Step or series of steps which operationally defines one component of the 
methodology. An. authoring tool is an evaluated, self-contained product in a suit- 
able format which is applied to the analysis, design, development, implementation, 
or control of one or more steps of a procedure. Authoring tools and procedures, 
of course, are used in a materials development cycle called the "authoring process.' 

The Interservice Procedures for Instructional Systems Development (IPISD) 
provided the model for ISD used in this study (TRADOC, 1975). The complete IPISD 
model is presented in Figure 1. 

2.2 METHOD 

^-^-l Search Strategy - Both American and Canadian literature from 1965 to 
the present were searched for references pertaining to instructional systems in the 
following areas: 

0 Developmental studies of on- and off-line instructional systems 

0 Evaluation and comparison of instructional materials development 
methodologies 

0 Design and installation of instructional materials 

0 "User studies" involving effectiveness of a medium and comparative 
effectiveness of alternative media 

0 Information processing behavior of students 

0 Analysis of training tasks and job requirements 

0 Procedures for revising instructional materials and systems, both on-line 
and off-line. 



Key words used in the literature search were 

0 Adaptive Models 

0 Adjunct Questions 

0 Advanced Organizers 

0 Audio-visual Materials 



0 
0 
0 
0 
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0 Behavioral Objectives o Programmed Instruction 

0 Branching o Programmed Texts 

0 Computer-Assisted Instruction o Remediation' 

0 Computer-Managed Instruction o Reviews 

0 Criterion-Referenced Testing o Scopes 

0 Del oyed Reviews o Summative Evaluation 

0 Educational Technology o Task Analysis 

0 Testing 

Computerized searches were made of data banks in the Educational Resources 
Information Center (ERIC), the National Technical Information Service (NTIS), the 
Defense Documentation Center (DDC), and the libraries of the McDonnell Douglas 
Corporation. Manual searches were made of indexes, abstracting services, and 
bibliographies at several local universities. In addition, a Selective Disserrnri- 
ation of Information (SDI) service was used along with personal communication with 
over seventy individuals and associations prominent in the field of instructional 
technology. Over 5,000 repo^'ts were read in full text or abstract form, and 
approximately 1 ,400 practical , development-oriented items were selected for 
inclusion in the computerized data base developed for the study. Approximately 
sixty documents were selected from the data base for inclusion in the annotated 
bibliography. Appendix B. 

2.2-2 Information Storage and Retrieval System - A computer-based information 
storage and retrieval system was established for the study. Hardware consisted of 
a teletype terminal connected via a communications link to a Xerox Sigma 9 computer. 
Printouts and formal reports were processed at a commercial computer-service office. 
Software consisted of two, high-level English language "packaged" programs: EDIT 
and CONFIRM. EDIT permitted easy data entry and allowed document titles to be 
accessed by keyword. CONFIRM was used for complex retrieval operations, statistical 
counts, and report generation. 

2.2.3 Selection Criteria - Authoring aids and information clearinghouses were 
classified as either tools, procedures, or methodologies used to analyze, design, 
develop, implement, and control ISD activities. Research reports and brochures 
from clearinghouses were selected for inclusion in the data base according to three 
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criteria. First, selected documents had to fit the definition of a tool, procedure, 
or methodology. Second, since the end result of the study was to be the identifi- 
cation and analysis of authoring aids usable in a military training environment, 
selected documents had to report work developed in a military setting, or in the 
civilian sector and be suitable for adaptation to operational military training. 
Third, work discussed in a report had to be applied and evaluated or, if a procedure 
or methodology, at least applied though unevaluated. Authoring methodologies were 
^ included in the study only for comparison with tools and procedures. Only one 
methodology was included in the annotated bibliography 

2-2 4 Procedure for Bibliographic Compilation - The procedure adopted for 
compiling the annotated bibliography consisted of the following four steps: 

^•^•^•^ De finition of Item Record - An item record was defined to include the 
characteristics by which research items could be compared. Data elements of the 
item record included: (a) personal or corporate author(s); (b) institutional 
affiliation; (c) publicatior, title; (d) pub ication date; (e) media; (f) type of 
application; (g) type of evaluation; (h) type of environment (military or civilian); 
and (i) type of ISD activity, according to the IPISD model. These data were entered 
by codes which allowed quicker, more accurate data entry and reduced item record 
length. For example, the institution, "University of Texas at Austin," was coded 
"B2480. " 



^•^•"^•^ Creation of Data Base - For documents read initially in full-text 
form, a complete set of data was input in one operation. Data entry for documents 
in abstract form required two steps. The symbol "ORD" (which indicated that an 
Item had been ordered) in the institution field and author and title were input 
to establish the document's item record. After the document had been received, it 
was analyzed to obtain the remaining data required, and the "ORD" code was replaced 
by the institutional affiliation code. Using "ORD" in the institution Tield auto- 
matically built an on-line order file which allowed the determination of the number 
of documents on order versus those on hand, and allowed the identification of 
excessive lags between orders and receipts as a basis for follow-up action A 
special -purpose program, called OLD TABLE, produced a daily total of item records 
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in each ISD activity and phase, and in the whole data base. Systematic use of this 
capability made it possible to "map" the data base and identify those ISD activities 
or phases requiring additional research. 

2.2.4.3 Analysis - The data base was analyzed in terms of authoring aids 
(tools, procedures, and methodologies) for developing instructional materials. 
Printout listings were obtained of research reports describing authoring aids that 
had been applied and evaluated in military or civilian environments. In some cases, 
an additional printout identifying unevaluated applications was also produced. 
Appropriate authoring aids were then selected from these listings based on three 
guidelines: large pools of applied and evaluated reports allowed some early or 
obsolete work to be rejected; relative cost and availability made some authoring 
aids more suitable than others; and approaches and concepts that had been widely 
used but not evaluated were occasionally chosen over narrower, more parochial work 
that had been evaluated. 

2-2.4.4 Cataloguing - The selected reports were categorized by the following 
considerations: 

a. Coded as a tool, procedure, or methodology, 

b. Coded by type of medium, application, evaluation, environment, and ISD 
activity. 
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. 3.0 RESULTS 

Three constraints limit the interpretation of the results. First, due to 
limited resources, classification and categorization of the reports surveyed were 
not subject to inter-judge reliability tests. Second, the population of reports 
in the data base was created by applying operational definitions of authoring tools, 
procedures, and methodologies, and the classification scheme to characterize the 
reports was derived by the author of this study. Third, though it may be suggested 
that documentation patterns for the total ISD R&D effort and this study may be 
similar (particularly for that portion of the effort applying to authoring tools 
and procedures), no specific evidence was systematically obtained to support this. 

3.1 RESEARCH HYPOTHESES 

To test the four hypotheses posed in Section 2.1, a total of 1424 documents 
were analyzed. Two pools of documents relative to ISD authoring aids were collected: 
(a) applied and eva' ;ated; and (b) applied but not evaluated. These documents were 
further classified by environment, either military or civilian, and by type of 
authoring aid. The numbers of documents in each classification are presented in ' 
Tables 1, 2, and 3. Chi square was used to test each research hypothesis. 

3.1.1 First Hypothesis - "The civilian environment has a greater number of 
applied, unevaluated documents than the military environment." The test of this 
hypothesis was statistically significant (X^ = 172.13, df = 1, p < .01), indicating 
that more applied, unevaluated documents were absociated with the civilian 
environment. (See Table 1) 

3-1-2 Second Hypothesis - "The civilian environment has a greater number of 
applied, evaluated documents than the military environment." The test of this 
hypothesis was not statistically significant (X^ = 1.73, df = 1, p > .01). This 
suggests that relatively equal numbers of applied, evaluated documents exist in 
each environment. (See Table 1) 

3.1.3 Third Hypothesis - "There are unequal numbers of documents within the 
three authoring aid categories." The test of this hypothesis was statistically 
significant (X^ = 145.72, df = 2, p < .01). Based on the raw data, it is apparent 
that more procedures were available than either tools or methodologies. (See Table 2) 
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3.1.4 Fourth Hypothesis - "The civilian environment has a greater number of 
authoring aids than the military environment." The test of this hypothesis only 
approached statistical significance (X^ = 6.26, df = 1 , .05 > p > .01 . Inspection 
of the raw data, however, suggests a tendency for the civilian environment to 
produce a greater number of authoring aids. (See Table 3) 

3.2 AVAILABILITY OF AUTHORING AIDS 

3-2.1 ISP Activities Lacking An Authorin g Aid - The raw data were analyzed 
to determine the ISO activities for which there were no tools. Tools are considered 
to be the most desirable authoring aid because they are readily transportable and 
require thi least adaptation or restructuring by potential users. No tools were 
found for the following ISO activities: 

0 Select Tasks/Functions 

0 Construct Job Performance Measures 

0 Select Instructional Settings 

0 Specify Learning Events/Activities 

0 Implement Instructional Management Plan 

0 Conduct External Evaluation. 

No authoring procedures were found for the ISD activities: 

0 Select Instructional Setting 

0 Specify Learning Events/Activities, 

3.2.2 Availability of Authoring Aids by ISD Activity - The steps of the 
IPISD model are specified in the fold-out in Appendix B. Each step has an input 
and an output. For example, one phase in the IPISD model is "Analyze." One 
step in that phase is "Analyze Job." One input to that step is a needs analysis 
imposed by installation of a new system, weapon, or equipment to a job or training 
situation. An output of that step is a validated list of job tasks concerning 
that new component. Although the output of one step may provide an input to the 
following step, the steps of the IPISD model are not necessarily linear or performed 
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STATUS 


ENVIRONMENT 




Military 


Civin'an 


Unevaluated 


57 


308 


Evaluated 


297 


331 


Total 


354 


639 



TABLE 1 : NUMBER OF APPLIED DOCUMENTS SURVEYED 



CATEGORY 


NUMBER 


Tools 


67 


Procedures 


260 


Methodoloqies 


104 


TABLE 2: NUMBER OF AUTHORING AIDS BY CATEGORY 

' ■' » 


ENVIRONMENT-- 


NUMBER 


Military 


189 


Civil ian 


242 



TABLE 3: NUMBERS OF AUTHORING AIDS BY ENVIRONMENT 
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on a once-and-once-only basis. The static model only provides a useful method for 
working on or talking c'J.out the dynamic process called ISD. Since the authoring , 
process is composed of four activities (information gathering, conversion, 
revision, and publication), authoring tools and procedures were drawn from ISD 
activities which analyzed, designed, developed, implemented, and controlled those 
activities. Available authoring tools and procedures that follow have been grouped 
according to ISD activities. Numbering of ISD activities is in accord with the ' 
fold-out cited above. 

I.l Analyze Job. An extensive Air Force occupational analysis program was ; 
begun in the 1960s that culminated in development of a very effective tool for 
computer-based job analysis. The CODAP system (for Comprehensive Occupational 
Data Analysis Programs) provides individual or group descriptions that specify 
the percent time spent on each task in a job, describes an actual job at different 
task levels such as apprentice or officer, derives individual and group job differ- 
ences, and identifies specific types of jobs as they exist or should exist within 
a general specialty. Detailed descriptions of procedures, programming notes, and 
applications are provided by Archer (1966), Morsch and Archer (1967), Christal 
(1972), Weismuller, Barton, and Rogers (1974a, 1974b), Stacey, Wei smuller. Barton : 
and Rogers (1974), Mayo, Nance, and Shiegekawa (1974), Stacey and Hazel (1974), 
and Christal and Weismuller (1975). 

Foley (1975) describes the use of several standardized worksheets for manual 
job analysis. The Task Inventory Exchange (TIE, 1975) collects occupational and 
task inventories and, to date, has published three task inventory directories. 

1-2 Select Tasks/Functions . McKnight and Adams (1970) report that a panel 
of experts selected critical behaviors for driver education training from a possible 
pool of 1000 relevant behaviors. Powers (1971) added a phrase to the standard task 
statement which aided the selection of tasks for teaching physician's assistants. 
The phrase, "in order to," was used to indicate an intended outcome of a task, and, • 
thereby, group students according to the required levels of competence. Cline (1973) 
used a multivariate statisticc' model to predetermine preferable aircraft assign- 
ments for Air Force pilots. 

1.3 Construct Job Performonce Measures . Gael (1974) describes the selection 
and design of instruments used to measure job aptitudes and job proficiencies for 
five Bell System jobs. The instruments employed both knowledge and performance 
measures as. well as simulation. 
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1.4 Analyze Existing Course. Lewis, Wniow, Brock, Lonigro, Eschenbrenner, 
*nd Hanson (1974) analyzed an Air Force course in Precision Measuring Equipment 
(Pf^E) I" convert it from conventional instruction to self-paced, individu- 

alized instruction. The analysis was done in four phases: (a) select course 
segments for individualization; (b) analyze course and job documentation; 
(c) choose the target population; and (d) examine the training environment. 
Formal analysis of an existing Air Force course appears in the Specialty Training 
Standard (STS), Plan of Instruction (POI), and the Course Chart for Inventory Manage 
ment Specialist (645X0) and Materiel Facilities Specialist (647X0). The documents, 
which provide codes for various levels of performance required at end of training 
and divide instruction into sequential segments, can be used to distinguish between 
required levels of performance (bgth cognitive and psychomotor) and to sequence 
instructfon accordingly. 

I 1.1 Develop Objectives- A programmed workbook intended as an introduction 
to behavioral objectives and to a Navy handbook Which teaches the writing of 
behavioral objectives is divided into three modules (Bureau of Naval Personnel, 
1968). Module 1 discusses the general concept of behavioral objectives. Module 2 
introduces the Navy's Handbook for Writing Learning Objectives , and Module 3 
contains an exercise kit for using the handbook and includes testing instruments. 
The Behavioral Objectives Training Package (ESCOE, 1971) serves as a user's guide 
to writing and implementing behavioral objectives and provides an overview of 
the operation of the Educational Service Center for Occupational' Education which 
contains a clearinghouse for behavioral objectives information. Oil Iman (1971) 
provides an excellent discussion of writing effective behavioral objectives. The 
discussion is presented in programmed-text format with self-reviews at the end of 
each section. Another programmed text by Rose, Balasek, Kelleher, Lutz and 
Nelken (1972a, 1972b) is designed to help teachers learn how to write performance 
objectives. An instructor's manual accompanying the programmed text provides 
management procedures for inservice training. Barton (1973) presents a booklet 
that teaches how to recognize and write technically correct performance objectives. 
Validated over a two-year period, the booklet is in programmed-text format and 
contains numerous self-checks. A compact, reuseable, multi-color Reference 
Wall Chart (INSGROUP, 1975) identifies the structure of measurable objectives, 
discusses three applications of measurable objectives, presents several taxonomies 
of learning which are used to analyze objectives, and contains a brief section of 
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annotations for the taxonomies. Also on the chart are three addresses where 
collections of objectives may be obtained. 

11.2 Develop Tests. Brown, Brooks, Cocks, and Kersch (1972) report a 
teacher workshop for defining, developing, and evaluating test items. The Center ' 
for the Study of Evaluation, in conjunction with Research for~Better Schools Inc., 
analyzed and evaluated approximately 2600 tests for measuring higher-order cognitive 
affective, and interpersonal skills (Center for the Study of Evaluation, 1972). 
A practical guide for preparing and using criterion-referenced tests in classroom 
instruction was developed by Gronlund (1.973). Checklists for evaluating criterion- 
referenced tests and procedures for item analysis are also given. Swezey and 
Pearlstein (1974) developed a manual that thoroughly covers the design, use and 
evaluation of criterion-referenced tests. Four steps are discussed: (a) develop 
adequate behavioral objectives; (b) develop a test plan and examine practical 
constraints; (c) build an item pool ; and (d) select the final test items for 
instruction. Stevens and O'Neil (1976a) discuss procedures for generating test 
items by item transformation, i.e., by varying sentences according to their 
syntactic structure. The test items can be used in multiple-choice or constructed 
response tests. Stevens and O'Neil (1976b, 1976c) suggest procedures for develop- 
ing and evaluating tests items for generalized multiple-choice tests and for 
comprehension-, application-, and analysis-level tests, 

II. 3 Describe Entry Behavior . Bierbaum and Planisek (1969) developed an 
index and procedure for determining thetprobable academic success of students 
seeking readmission after academic dismissal. An interest inventory designed to 
measure the vocational interests of enlisted men entering the Air Force was 
developed by Ecternacht, Reilly, and McCaffrey (1973). Called VOICE (for Vocational 
and Occupational Interest Choice Examination), the instrument was to be used in 
the development of an interest inventory for the Guaranteed Enlistment Program. 
Mockovak (1974) assessed the literacy requirements of Air Force career ladders 
in order to determine the reading demands placed upon trainees and job incumbents 
by instructional materials. Vitola, Mullins, and Brokaw (1973) found that removal 
of the draft as a motivator for enlistment affected the Air Force in three ways: 
(a) decreased accessions completing education beyond the high school level; (b) . 
the discharge rate for first-term airmen after four years was 79 percent; and (c) 
the average difference on mean scores for AQE apcitude composites between Blacks 
and Non-blacks decreased. 
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^^•^ Determine Sequence and Structure. Dansereau, Evans, Wright, Long, and 
Actkinson (1974) and Dansereau, Evans, Actkinson, and Long (1974) developed and 
°^ scaling (called INSCAL) as a procedure for 
describing the information structure of Air Force instructional materials and 
generating effective instructional sequences. A checklist for determining 
students' positions along a curriculum scale was developed (Bucks County Public 
Schools, 1970) which may be useful as a manual backup to computer-based learning 
systems for assessing student performance and determining proper learning 
sequences. 

III. 3 Review/Select Existing Materials . The State University College at 
Buffalo (1972) has , published a user's manual which describes computer-based resource 
units in various subject areas and gives directions for obtaining them. Up to five 
units may be requested, and individual difference variables may be indicated on the 
request form. An annotated listing of curriculum materials which can be obtained 
from state education agencies is available from the U.S. Government Printing 
Office (1973). Various guides cover agriculture, distributive education, health 
occupations, home economics, technical education, and trade and industrial occupa- 

The National Ne twork for Curriculum Coordination in Vocational-Technical 
Education develops, publishes, and distributes curriculum guides and various 
curriculum support services from six regional curriculum management centers. 
Curriculum guides are published in the form of brochures, booklets, and microfiche. 
"•"•^^ Special Reserve Components Educational Video Tape Catalog contains a listing 
of educational video tape programs selected for Army unit training. Though the 
materials are intended for Reserve component use, they may be used by other Services, 
Army schools, and training centers. The video tapes may be- borrowed from various 
Training Aids Support Offices (TASOs) or can be copied on user-supplied cassette 
tapes. The Directorate for Audio-Visual Activities (DAVA, 1975) presents'a detailed 
user's guide that provides the required information for participating in the 
Defense Audio-Visual Information Network and for utilizing the network's Audio- 
Visual Products Data Base. Appendices give input data guidelines for 
data coding and other computer-related activities. The Department of the Army 
(1974, 1975) provides two catalogs of Army motion picture films and audio-visual 
aids. The first catalog gives annotated listings of motion picture films, slides, 
audio-visual kits, recordings, and tape recordings intended for use by instructors, 
information officers, and defense contractors. The second catalog provides annota- 
tions of motion picture films for non-profit use by Government agencies, civic, 
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religious, fraternal and educational organizations, schools, colleges, and uni- 
versities. Materials from the first catalog are oriented to Army career fields, 
while those in the second catalog are on more general subjects such as education. , 
sports, and health. 

II 1.4 Develop Instruction. Best (1971) discusses various procedures and 
materials which minimize expense and time in the preparation and production of 
computer-controlled films. For local production shops, this report provides much 
ot the information required for processing filmstrips into slides for either 
computer-controlled, rear-screen projectors or for transparencies. Thiagarajan 
(1971) offers a programmed text for teaching the craft of programming. Five 
stages of the programming process are discussed, and a confirmation section at 
the conclusion of the book contains correct answers to the questions embedded in 
the text. How to Design and Produce Individualized Instruction Programs 
(Minnesota Mining and Manufacturing Co.. 1972) is a pragmatic workbook 
intended to be used for developing audio-visual programs for the 3M Sound-On- 
Slide System. In another user's manual, guidelines for writing computer-based 
resource units are presented and organized into two operations (State University 
of New York. 1972). The first operation involves formulating behavioral objectives, 
instructional content, activities, materials, and measuring devices. The second 
operation involves coding these five component items for future computer retrieval. 
Deterlirie and Lenn (1972) developed a self-study course that teaches a set of 
skills called CISTRAIN (for Coordinated Instructional System for Traini ng) for 
designing, developing, and implementing instruction. The two-volume set of 
instruction includes a Lesson Book and a'study Resource Materials Book. Siegel. 
Lambert, and Burkett (1974) present a concise, entertaining synthesis of readability 
research and illustrate findings from that field that are useful to course writers. 
Nesbit and O'Neil (1976) provide guidelines for editing programmed texts. The 
report contains a glossary of terms, explanations of three models for programmed 
texts, and guidelines for checking*the lesson content and author's style. 

III-5 Validate Instruction. The use of computer-based test analyses and course 
evaluation questionnaires is reported by Aleamoni (1970). The system provides for: 
(a) test scores; (b) analysis of scores for groups or individuals; (c) test 
item analysis printouts; (d) item analysis data; and (e) analysis of course 
evaluation questionnaires, attitude scales, and other measures which have no 
known correct response. A manual and system documentation are provided. Abedor 
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(1972) presents a formative evaluation model which functions in a real-world, 
operational environment. Noting that multiple, iterative revisions of materials 
can be a monumental and costly effort. Abedor designed a formative evaluation 
model to generate large amounts of data on instructional problems in a one-shot 
trial of the prototype lesson. Ward (1972) investigated an author's use of forma- 
tive evaluation during the preparation of a CAI course. Rayner (1972) developed 
a formative evaluation model that provides data not only for content but also for 
procedural changes to instructional materials. Hummel-Rossi (1972) devised a 
multiple-choice test for evaluating knowledge of Bloom's taxonomy. The test 
includes an instruction sheet, a specially-treated answer sheet which, when . 
rubbed, displays the answer, and the testing instrument. 

V.l Conduct Internal Evaluation . Bailey (1972) discusses the testing of 
manual procedures in the Bell System personnel subsystem. The procedures consist 
of six steps and provide data concerning the numbers and kinds of errors and 
the probable causes of errors for various job activities. Miller and Sellman 
(1973) developed a new student critique form for the Air Training Command which 
.indicated that. (a) attitudes of trainees toward technical training were "slightly 
positive;" (b) the most positive attitudes toward technical training were held 
by Non-Commissioned Officers, followed by Officers, and then Airmen; and (c) 
airmen were uncertain that technical training was adequately preparing them for 
on-the-job performance. 

V.2 Conduct External Evaluation. Anastasio (1972) proposed a field-test 
evaluation of the PLATO and TICCIT systems that included cost and educational 
analyses. Of interest to course writers are the discussions of achievement and 
attitude measures and the Survey of Instructor Activities and Attitudes contained 
in Appendix A of the Anastasio report. 
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4.0 CONCLUSIONS AND RECOMMENDATIONS 

The preceding findings suggest four areas for further consideration. First, 
an ISD activity such as Select Instructional Setting that has no tool or procedure 
associated with it should be investigated to determine whether a tool or procedure 
is critical to the role of that ISD activity in education or training. If a tool 
or procedure is critical to the execution of that activity, further investigation 
should be conducted in order to create the tool or procedure. 

Second, if a tool is generally considered the most useful authoring aid for 
an ISD activity, then the proportion of procedures identified in the present study 
should be modified into tools. 

Third, in an ISD activity where several tools and procedures but few method-, 
ologies are available, priority should be given to further research and development. 
In test development, for example, nineteen tools and procedures were identified for 
constructing criterion-referenced tests which are currently the most widely-used 
form of testing instrument.. However, test development is identified with only six 
methodologies. When new situations arise where criterion-referenced testing is 
not appropriate and few test development methodologies exist, an unacceptable lag. 
between formulation of new testing systems may occur. 

Fourth, during the course of the study, six significant aspects of the author- 
ing process were identified: (a) size and configuration of^authoring teams; (b) 
knowledge and proficiency of course authors; (c) existing course structures 
including lesson materials and course control documents; (d) type of media and 
types of instruction; (e) extent and type of evaluation and revision; and (f) text- 
handling and reproduction systems. The only subjective aspects of the authoring 
process are knowledge and proficiency of course authors. Since identification 
and distribution of authoring tools and procedures would allow course authors to 
develop materials from a baseline of equal knowledge, the objective aspects of the 
author process can be categorized and manipulated in a ^'laboratory" (i.e., con- 
trolled) setting. Thus, it should be possible to model authoring process variables 
in order to formulate decision rules for managing the materials development cycle 
with an optimum mix of personnel and organization, 
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In summary, ISD has been discussed as an adaptation of the systems approach. 
The product of an ISD effort was seen as an instructional system, one component 
of which was the authoring process. Four activities of the authoring process 
were identified: information gathering, conversion, revision, and publication. 
Authoring tools and procedures were identified for each of the four activities. 
It was recommended that an authoring process model be developed for formulating 
decision rules for properly managing development of instructional materials. 
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APPENDIX A 

Data Item Names and Codes for Document Entries 



DATA ITEM NAME 



DATA ITEM CODE 



Author- 
Institution 



Title 

Date of Publication 
ISD Activity Number 
Type of Media 



Type of Application 



Type of Evaluation 
Type of Environment 
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Self-explanatory 

B = College or University^ • 

Y = Non-profit Organization 

Z = Profit-making Organizatioti' 

Self -Explanatory 

S e 1 f - Ex p 1 a n a 1 0 ry - 

0 = ETV 

1 = Radio 

2 = Programmed Text 

3 = CAI 

4 = CMI 

5 = CAI/CMI 

6 = Audio-visual 

7 = Lecture 

8 =. Other 

0 = Theory 

1 = Applied 

2 T Applied in Locks tep 

3 = Applied in Self -paced 

0 = Evaluated 

1 = Not evaluated 

0 = Civilian 

1 = Joint Military Service 

2 = Army 

3 = Navy 

4 = Air Force 

5 = Marines 
6 '=^Nbrie^^^ 
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APPENDIX A 
(Continued) 



^College and university codes were followed by a four-digjt numeric 
code obtained from the Skills Inventory System Code Book (St. Louis: 
McDonnell Douglas Corp., 1974). Profit and non-profit-making organiza- 
tion codes were followed by that organization's accepted abbreviation. 
For example, ZIBM indicated the International Business Machines Corpora- 
tion and YYMCA indicated the Young Menu's Christian Association. Where 
no standard abbreviation could be obtained, only the one-letter code , 
was used. 



ISD activity numbers follow those of the Interservice Procedures for 
Instructional Systems Development Model shown in Figure 1 . For example, 
2.2 indicates "Develop Tests." 
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APPENDIX B 
AUTHORING AID ANNOTATED BIBLIOGRAPHY 

The abstracts in the bibliography are grouped in accordance with the steps$-''of 
the IPISD model. The IPISD model consists of five phases and nineteen steps^ the 
phases are Analysis, Design, Development, Implementation, and Control .^.'H'he steps 
within each phase are summarized below. 

y 

■ 

Phase I. Analyze . A certain training or educational program is determined 
to be both desirable and feasible. The job or preferred outcome to be' taught is 
analyzed as a series of task statements'. Numerical, analytic techniques and the 
judgment of subject matter experts are used to select a subset of the task state- 
ments for training and instruction. The instructional setting, i.e., whether the 
training and instruction should be conducted in formal schools, on job situations, 
in correspondence courses, or wherever, is determined. 

Phase II. Design . The tasks selected for training and instruction in Phase 
I are converted into behavioral objectives which specify the outcomes to be obtained 
at the end of training. Criterion measures are devised for measuring learner per- 
formance (both cognitive and psychomotor, if desired) against the objectives. 
Incoming student characteristics are identified in terms of general ability, reading 
level, and prior experience. 

Phase III. Development . Carefully selected and prepared media and materials 
are developed. Decisions are made concerning (a) management of students, instruc- . 
tors, and course; (b) desired learning experiences and activities; and (c) the 
form and content of the instructional delivery system. Instructional materials 
are developed and validated during small- and large-group tryouts. Instructional 
materials and procedures are revised in order to assure that learners meet lesson 
and course objectives. 

Phase IV. Implementation . Instructional and management staff are trained 
to meet the requirements of the instructional management plan and delivery system 
selected as the instructional vehicle. Media and materials are implemented in 
approved ,v large-scal e courses of instruction , and feedback"ls- obtained "on b 
system performance and effectiveness of materials. 



SH.IInl^D.'iSl'' "fC 61570 

BIBLIOGRAPHY VOLUME I 11 FEBRUARY 1977 

Phase V. Control . Data obtained during instruction are used to answer two 
questions. First, are students learning what they were intended to learn; and 
second, did what the students learn benefit them after training, i.e., during 
later employment or further study? Negative answers to either question, which 
are identified during internal and external (or field) evaluations, indicate that 
the contents and procedures of the new system should be revised. 

Each abstract in the bibliography is prefaced by a three-part number and a 
step title. The reference number on the left side of the page indicates the ISD 
phase, the step in that phase, and the particular abstract in that step. The step 
title on the right side of the page provides an easy reference point for the reader. 
For example, an abstract prefaced with the number "3.1.2" indicates that the 
abstract is concerned with Phase 3 (Development), Step 1 (Developing Objectives) 
and that it is the second abstract in that step: 



Phase 3 ^ Step 1 ^ ^ Abstract #2^ 

The step title translates the code "3.1" into "Specify Learning Events/Activities." 
To retrieve information concerning a particular ISD activity or step, merely con- 
sult the Table of Contents of the bibliography and turn to the section of abstracts 
beginning with the appropriate Phase/Step number. The abstracts that follow 
begin with Phase 1, Step 1, Abstract 1 (1.1.1) and continue through with Phase 5, 
Step 3, Abstract 2 (5.3.2). Refer to the fold-out on the following page as neces- 
sary for identification and sequence of ISD activities and steps. 
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PHASE 1 
ANALYZE 



Inputs, processes, and outputs in Phase 1 are all based on job information. An inventory of job 
tasks IS compiled and divided into two groups: tasks not selected tor instruction and tasks 
selected for instruction. Performance standards for tasks selected for instruction are determined 
by interview or observation at job sites and verified by subject matter experts. The analysis of 
existing course documentation is done to determine if all or portions of the analysis phase and 
other phases have already been done by someone else following the ISD guidelines. As a final 
analysis phase step, the list of tasks selected for instruction is analyzed for the most suitable 
instructional setting for each task. 



PHASE 2 
DESIGN 



Beginning with Phase 2, the ISD model is concerned with designing instruction using the job 
analysis information from Phase }. The first step is the conversion of each task selected for 
training into a terminal learning objective. Each terminal learning objective is then analyzed to 
determine learning objectives and learning steps necessary for mastery of the terminal learning 
objective. Tests are designed to match the learning objectives. Asample of students is tested to 
insure that their entry behaviors match the level of learning analysis. Finally, a sequence of 
instruction is designed for the learning objectives. • 



PHASE 3 
DEVELOP 



The instructional development phase begins with the classification of learning objectives by 
■ learning category so as to identify learning guidelines necessary for optimum learning to take 
place. Determining how instruction is to be packaged and presented to the student is 
accomplished through a media selection process which takes into account such factors as 
earning category and guideline, media characteristics, training setting criteria and costs 
Instructional management plans are developed to allocate and manage all resources for 
conductiog instruction. Instructional materials are selected or developed and tried out When 
materials have been validated on the basis of empirical data obtained from groups of typical 
students, the course is ready for implementation. 



Staf training is required for the implementation of the instructional management plan and the 
PHASE 4 ZTtT' ^""^^."^^V -^"^t be trained to be managers in the specified management 

IKiin. rr»V?^ll H , -^"^ be trained to conduct the instruction and collect evaluative 

IMPLEMENT ''^'^ °" °' instructional components. At the completion of each instructional cycle 
management staff should be able to use the collected information to improve the instructional 



PHASES 
CONTROL 



Evaluation and revision of instruction are carried out by personnel who preferably are neither 
the instructional designers nor the managers of the course under study. The first activity 
(internal evaluation) is the analysis of leafner performance in the course to determine instances 
of deficient or irrelevant instruction. The evaluation team then suggests solutions for the 
problems. In the external evaluation; personnel assess job task performance-dnnhejaFTT' 
determine the actual performance of course graduates and other job incumbents. All collected 
aata, internal and external, can be used as quality control on instruction and as input to any 
phase of the system for revision. 
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THE BLOCKS IN EACH PHASE ARE: 



1.1 

►ANALYZE 
JOB 



1.2 
^SELECT 
^ TASKS/ 

FUNCTIONS 



1.3 

CONSTRUCT 
► JOB 
PERFORMANCE 
MEASURES 



1.4 

.ANALYZE 
^EXISTING 
COURSES 



1.5 
.SELECT 

^NSTRUCTIONAL 
SETTING 



2.1 
►DEVELOP 
OBJECTIVES 



2.2 
►DEVELOP 
TESTS 



2.3 
.DESCRIBE 
ENTRY 
BEHAVIOR 



2.4 

DETERMINE 
SEQUENCE & 
STRUCTURE 



11 

SPECIFY . 
► LEARNING 
EVENTS/ 
ACTIVITIES 



3.2 

SPECIFY 
INSTRUCTION 
> MANAGEMENT 
PLAN & 
DELIVERY 
SYSTEM 



3.3 

.REVIEW/SELECT 
EXISTING 
MATERIALS 



3.4 
►DEVELOP 
INSTRUCTION 



3.5 

► VALIDATE 
INSTRUCTION 



4.1 

IMPLEMENT 
► INSTRUCTIONAL 
MANAGEMENT 
PLAN 



4.2 
► CONDUCT 
INSTRUCTION 



5.1 

^ CONDUCT 
INTERNAL 
EVALUATION 



5.2 
.CONDUCT 
EXTERNAL 
EVALUATION 



5.3 
► REVISE 
SYSTEM 



THE OUTCOMES OF THE BLOCKS ARE: 



,1 ... a list of tasks performed in a particular job. 
.2. .. a list of tasks selected for training. 

.3. . . a job performance measure for each task selected for instruction. 

,4. . . an analysis of the job analysis, task selection, and performance measure 

construction for any existing instruction to determine if these courses are 

usable in whole or in part 

.5. . . selection of the instructional setting for task selected for instruction. 



,1 ... a learning objective for and a learning analysis of each task selected for 
instruction. 

.2 . . . test items to measure each learning objective. 

.3 ... a test of entry behaviors to see if the original assumptions were correct. 
.4. . . the sequencing of all dependent tasks. 



,1 . . . the classification of learning objectives by learning category and the 
identification of appropriate learning guidelines. 

,2... the media selections for instructional development and the instructional 
management plan for conducting the instruction. 

.3... the analysis of packages of any existing instruction that meets the given 
learning objectives. 

,4... the development of instruction for all learning objectives where existing 
materials are not available. ' 

,5. . . field tested and revised instructional nriaterials. 



,1 . , . documents containing information on 
resources, and staff trained to conduct 

,2. . . a completed cycle of instruction with 
the succeeding cycle. 



time, space, student and instructional 
the instruction. 

information needed to improve it for 



,1 . . , data on instructional effectiveness. 
,2. . . data on job performance in the field. 
,3. . . instructional system revised ori bas^^^^^^ 
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REFERENCE NUMBER: 1.1.1 ANALYZE JOB 

CITATIONS: 



.. Archer, W. B. Computation of group~job descri"priTnTTroFocc"up^^ 

(Report No. PRL-TR-66-12); Lackland Air Force Base, Texas: Air Force Systems 
Command, Aerospace Medical Division, Personnel Research Laboratory, December 
1966. 

Morsch, J. E., & Archer, W. B. Procedural guide for conducting occupational surveys 
in the United States Air Force (Report No. PRL-67-11 ). Lackland Air Force Base, 
Texas: Air Force Systems Command, Aerospace Medical Division, Personnel 
. Research Laboratory, September 1967. 

Christal , R. E. CODAP: Input standard (INPSTD) and variable generation (VARGEN) 
programs (Report No. AFHRL-TR-72-51). Lackland Air Force Base, Texas:. Air 
Force Human Resources Laboratory, Personnel Research Division, May 1972. 



Stacey, W. D., Weissmujler, J. J., Barton, B. B. , & Rogers, C. R. CODAP: Control 
card specifications for the UNIVAC 1108 (Report No. AFHRL-TR-74-84). Lackland 
Air Force Base, Texas: Air Force Human Resources Laboratory, Computational 
Sciences Division, October 1974. 

Weissmuller, J. J., Barton, B. B., & Rogers, C. R. CODAP: Source program listings 
for the UNIVAC 1108 (Report No. AFHRL-TR-74-R3L Lackland Air Force Base, 
Texas: Air Force Human Resources Laboratory, Computational Sciences Division, 
October 1974a. 

Weissmuller, J. J., Barton, B. B., & Rogers, C. R. CODAP: Programmer notes for 
the subroutine library on the UNIVAC 1108 (Report No. AFHRL-TR-74-85). 
Lackland Air Force Base, Texas: Air Force Human Resources Laboratory, Computa- 
tional Sciences Division, October 1974b. 

Mayo, C. C, Nance, D. M., & Shigekawa, L. Evaluation of the job inventory 

approach in analyzing USAF officer utilization fields (Report No. AFHRL-TR-75- 
22). Lackland Air Force Base, Texas: Air Force Human Resources Laboratory, 

Occupational and Manpower Research Di vi sion "Dune 1 975. 
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Stacy, W. J., & Hazel, J. T. A method of determining desirable task experiences for 
first-line supervisors (Report No.. AFHRL-TR-75-23). Lackland Air Force Base, 
Texas: Air Force Human Resources Laboratory, Occupational and Manpower 
Research Division, August 1975. 

Christal, E. E., & Weissmuller, J.'O. New CODAP programs for analyzing task factor 
information. Paper presented at the meeting of the Military Testing Association, 
Indianapolis, September 1975. 

ABSTRACT: 

An extensive Air Force occupational analysis program was begun inthe 1960s that 
culminated in development of an on-going computer-based occupational analysis sys- 
tem. Providing valid job analysis information, the Comprehensive Occupational JData 
Analysis Programs (CODAP) allow job analysts to describe and evaluate existing jobs 
within Air; Force Career Specialties, and to establish training priorities for new 
courses. The CODAP. systeifi^can: (a) provide individual or group descriptions that 
specify the percent time spent on each task in a job; (b) describe an actual job at 
different task levels (such as apprentice, officer, and so forth); (c) derive indi- 
vidual and group job differences; and (d) identify specific types of jobs as they 
exist or should exist within a general specialty. These features, as well as an 
introduction and program notes, are described in the series of citations discussed 
below. 

Archer (1966) provides a detailed analysis of occupational survey data by using 
a miniature sample. Based on responses of ten incumbents to a job inventory con- 
sisting of ten task statements, composite job descriptions are derived for (a) 
special groups of incumbents, selected on the basis of background information data; 
and (b) job type members, identified by an automated job clustering program. Illus-: 
trations of computer outputs from both types of analysis are explained. The 10 
incumbents are Medical Service personnel with duty level ratings of three, five, and 
seven. The ten task statements are concerned with supervising subordinates, perform- 
ing ward procedures, and performing outpatient procedures. Each incumbent filled 
out a job inventory survey by checking the tasks he performed as part of his job 
and rating the tasks on a nine-point relative timerspent scale. In a very detailed 
discussion with 38 tables and four figures, the author illustrates how task-and- 
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duty-level descriptions are derived, af>d how membership and homogeneity of job types 
are clustered. 

Morsch and Archer (1967) describe the procedures for collecting, organizing, 

analyzing, and reporting j^o task information of Air Force officers and airmen. 
Discrete steps in the application of the Air Force job analysis method are given in 
the following chronological order: (a) location and procurement of source materials; 
(b) construction of preliminary job inventory; (c) internal review of preliminary 
job inventory; (d) revision of preliminary job inventory} (e) field test/review of 
revised job inventory; (f) construction of final job inventory; (g) selection and 
location of occupational survey samples; (h) reproduction of job inventory materials 
(i) packaging and mailing job inventories'; (j) administration of job inventories ; 
(k) receiving, scanning, and coding job inventory booklets; (1) integrating added 
job inventory information; (m) keypunching and verifying job inventory data; (n) 
computer analysis of job inventory data; and (o) the Occupational Survey Report. 
With cl^ar. concise discussion of each step plus fifteen appendices which contain 
everythiit:' te* instructions for self- or group-administration of the job inventory 
to a sample letter of appreciation, this report is an evaluated, self-contained tool 
for job or task analyzers. 

The next four reports present information to be used by computer specialists 
aiding the job analysis effort. Christal (1972) describes and gives application 
examples of CODAP's Input Standard Program (INPSTD) and Variable Generation Program 
(VARGEN). Stacey, Weissmuller, Barton, and Rogers (1974) present control card 
write-ups for 34 main programs in the UNIVAC 1108 version of CODAP. Arranged 
alphabetically by program name and preceded by a short summary of the program" 
functions, each write-up contains: (a) detailed control card specifications neces- 
sary for using the CODAP system; (b) gencr.ii input-output specifications; (c) general 
program logic; and (d) diagnostic aids. Weissmuller, Barton, and Rogers (1974a) 
cover source programs for the UNIVAC 1108 version of CODAP. Included are 34 FORTRAN 
main programs and 78 FORTRAN and 20 Assembly Language subroutines. Weissmuller, 
Barton, and Rogers (1974b) discuss programmer notes on 100 library subroutines used 
by the UNIVAC 1108 version of CODAP. Arranged in alphabetical order by subroutine 
name, each note includes: (a) comments on peculiar requirements for conversion and 
use of subroutiniss; (b) summaries of subroutine functions and a list of entry 
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points; and (c) for each entry point, a calling sequence with descriptions of 
input and outpift arguments. , _ 

Typical uses of the CODAP system for course developers- are given in the next two 
reports. First, the job inventory method of job analysis for officer positions was 
studied by Mayo, Nance, and Shigekawa (1974). The authors concluded that the in- 
ventory method was operationally adequate but, due to the officers' broader, 
concept-oriented responsibilities, more specific approaches to task construction are 
required than are usually needed for airmen positions. To resolve the issues of 
task specificity and breadth of coverage for officers' fields, "heavy front-end 
work" (i.e., research requisite information) is recommended. 

Second, skill-level career ladders in Accounting and Disbursement were studied 
by Stacey and Hazel (1974). More than 1200 Accounting and Finance Supervisors 
examined 254 journeyman-level tasks and rated the importance of each task relative 
to satisfactory supervisory performance. Job analyses of career work experiences 
indicated: (a) many supervisors had limited career experience in the desirable 
journeyman-level tasks; (b) many supervisors had spent most of their career time in 
very few of eight accounting and finance work areas;. and (c) lack of supervisory 
work experience was related to on-the-job performance. The authors concluded that 
the present accounting and finance work management unit was too broad for required 
work activities, and recommended the separation of the Accounting (671x1) and Dis- 
bursement (671x3) career ladder through the seven-skill level. 

In a recent paper, Christal and Weissmuller (1975) describe five new programs 
added to the UNIVAC 1108 version of CODAP whicH are designed to analyze and manipu- 
late information describing work tasks rather than jobs or- persons. One use of the 
new program is to rank-order training priorities for curriculum designers. Although 
the technical discussion is beyond the intent of this bibliography, the authors re- 
port that control card specifications are now available which can be reprogrammed 
for execution on IBM 360-370 series computers. 
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DATA SUMMARY: 

Media: Computer- hased 

Application: Computer-based job analysis 

Evaluation: Extensive field-tests 

Environment: Military (Air Force) 

Key Number: 1.1 Analyze Job 
Authoring aid type: Tool 
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REFERENCE NUMBER: 1.1.2 ANALYZE JOB 

CITATION: 

Foley, J. P., Jr. Task analyses for job performance aids and related training. In 
T. C. Rowan (Ed.), Proceedings of the Invitational Conference On Improved - 
Informati on Aids for Technicians . Sumner, Maryland: Logistics Management 
Institute, May 1975. 

ABSTRACT: 

A major problem in educating instructional personnel in the use of task analysis 
has been a lack of agreement as to its purpose and method of execution. Emphasizing 
maintenance, Foley argues that perhaps the more important purpose of task analysis is 
to determine the content of training, and offers a standardized execution methodology. 

The first step in task analysis is, of course, identification of tasks actually 
done on the job. If job requirements are being done only partially, then what peo- 
ple should do should also be identified. Work sheets for a Task Identification 
Matrix (TIM) are given which list each system hardware component and its parts on a 
numbered line. Column headings on the TIM consist of 14 maintenance actions such 
as adjust, align, calibrate, checkout, handle, clean, lubricate, and inspect. Then, 
for each cell in the matrix, a code is entered which indicates the level of mainte- : 
"3nce required. After every task has been identified and scoped, a Task Inventory 
is produced which is simply a listing and numbering of each task identified in the 
matrix. The computer storage and handling of task information would permit auto- 
matic generation of task inventories. (See references for the CODAP System, 1.1.1) 

The second step in task analysis is functional analysis of the identified tasks. 
Under the umbrella of a Task Description Index and Management Matrix (TDIMM), the 
functional analysis information is formatted on three worksheets. The preliminary 
Information. Worksheet lists the hardware item with which the task is associated as 
well as required supplies, forms, replacement parts and technical orders, and neces- 
sary notes, cautions, and warnings. The Task Description and Information Index 
merges this information with summary information about the task such as criticality, 
frequency of occurrence, number of steps, and task type. The Detailed Step Descrip- 
tion Worksheet provides for a detailed, step-by-step analysis of. the task. It in- 
cludes maintenance support information and special instructions. When a task 
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involves use of test equipment and special tools, a Test Equipment and Topi Use 
Form is completed. Graphics of each form are given in the text. 

Using these documents does not ensure that effective directions are finally 
written for the maintenance technician. A third step is required which involves 
"a software middle man" who is able to add the proper behavioral cues to the techni- 
cal directions and analysis. For example, in one application where exploded views 
of an aircraft did not supply the required pictorial cues for disassembly, Foley 
says that it was necessary to add the phrase "when lens is closed it looks shiny." 

A variant of conventional task analysis is special analysis for troubleshooting 
tasks. Special guidance must be given both to task analyzers and technicians. 
Foley recommends the use of Maintenance Dependency Charts, Logic Trees, and Fully 
Procedural i zed Job Performance Aids, details of which are beyond the scope of his 
report. 

Although the author used maintenance examples to illustrate the utility of 
standardized worksheets, it should not be difficult to adapt the forms for use in 
other task analysis situations. 

DATA SUMMARY: 

Media: Print 

Application: Manual task analysis 

Evaluation: None 

Environment: Military (Arrny) 

Key Number: 1.1 Analyze Job 

Authoring aid type: Tool 

Number of pages: 21 ^ • 

Number of references: 31 
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REFERENCE NUMBER: 1.1.3 ANALYZE JOB 

CITATION: 

Task Inventory Exchange. Brochure . Columbus: The Ohio State University, Center 
for Vocational Education, 1975. 

ABSTRACT 

Task analysis is a time- and money-consuming activity which generates task 
inventories for specific jobs in organizations or occupational areas with a limited 
intent of use on the part of their developers. However, if their existence and 
character are known these same job inventories may have much wider application. To 
bridge the gap between those who need but are unaware of existing task inventories 
and those who have previously developed them, the Task Inventory Exchange (TIE) was 
created. The TIE has four functions: (a) identify existing occupational/task in- 
ventories; (b) collect, catalog, and store existing task inventories; (c) publish 
directories of available task inventories; and (d) supply microfiche or hard copies 
of available task inventories.. 

The TIE obtains task inventories through distribution of a brochure and place- 
ment of advertisements in eight professional journals. Contributions of task in- 
ventories and related literature for the TIE should be made in two copies. It also 
^^^iX^^-y c°'^P''^'^s"siye literature searches in the data base^^ 

Resources Information Centers (ERIC), National Technical Information Service (NTIS), 
Abstracts of Instructional/Research Materials in Vocational and Technical Education 
(AIM/ARM), and University Microftlms. Results of these activities are collected and 
stored on magnetic tape and in hard- copy volumes. 

The TIE periodically publishes directories of task inventories. Volume 1, 
Directory of Task Inventories , was published in 1974 and contained citations for 
.77 publications listing task inventories for 280 job titles or occupational areas. 
Volume 2, same title, was published in 1975 and contained citations for 177 publica- 
tions listing inventories for 379 occupations or jobs. Volume 3 is currently in 
press. The cost for Volumes 1 and 2 is $4 and $7, respectively. 

The TIE supplies partial or whole task inventories on a cost- recovery basis. 
The first five microfiche of any one document are 65 cents and each additional 
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fiche is 15 cents. Paper copies of task inventories are five cents a page. 

Further information concerning the TIE may be obtained from: 

Paul E. Schroeder, Project Director 

Task Inventory Exchange 

Center for Vocational Education 

The Ohio State University 

1960 Kenny Road 

Columbus, Ohio 43210 

(614) 486-3655. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key Number: 
Authoring Aid Type: 
Number of Pages: 
NumLjr of References: 



Microfiche, Print 
Job Analysis 
None 

Civilian 

1.1 Analyze Job 

Tool 

Approximately 20 
4 
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REFERENCE NUMBER: 1.2.1 SELECT TASKS/ FUNCTIONS 

CITATION; ' ■ 

Cline, J. A. A multivariate statistica l model to predetermine preferable aircraft 
assignments: A feasibility study (Doctoral Dissertation, Arizona State Univer- 
sity, 1973). University Microfilms No. 73-20, 494. 

ABSTRACT: 

If job assitnments can be made before students begin or complete training,^ it 
is possible to identify appropriate training tasks or functions that are needed for 
job assignment and tailor individual learning programs for individual students. 
Cline developed a model for selecting pilots for end aircraft assignment (advanced 
pilot training) early in Undergraduate Pilot Training. His technique may be ex- 
tended to other fields. 

Peer-ranking data were obtained from an earlier study and used as the primary 
operational measure of a pilot's success flying an operational aircraft. Secondary 
measures included advanced pilot training grades, and on-the-job Officer Effective- 
ness Ratings by supervisors. Two techniques used in the study are of interest to 
course developers: (a) conversion of peer-rankings into ordinal dataj and (b) selec- 
tion of subgroups for classes of training. 



ERJC 



One disadvantage of peer ranking is that while it is possible to determine who 
is first, second, or third, it is not possible to measure the amount of .ciifference 
'between the first and the second, and between the second and the third. / Therefore, 
Cline converts the peer rankings into ordinal data by using a technique," similar to 
Thorndike's T-Score procedure. The ordinal data become the criteria for the further 
selection of subgroups for classes of training. The validity of the criteria are. 
Cline admits, subjective and "cannot be statistically measured", but their relia- ' 
bility was measured by the coefficient of concordance (W), which is a measure of 
the commonality of judgements. In this particular study, it indicated the degree 
to which pilots kept the same relative standing in the group when peers ranked all 
members of the group. 

Selection of subgroups for classes of training was based upon selected variables 
such as military grade and flying hours in jet-powered aircraft, and upon a 
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Statistical technique called Automatic Interaction Detector (AID). Developed and 
refined by Air Force personnel, the AID technique essentially divides the total 
sample into mutually exclusive subgroups of increased homogeneity with respect to • 
the dependent variable. For example, using AID, Cline found that the most important 
variable for explaining the variance in pilot Armed Forces Qualifying Test (AFQT) 
was education. The first split separated college graduates from high and non-high 
school graduates and accounted for 68 percent of the variance. The second split 
separated high and non-high school graduates and accounted for an additional nine 
percent of the variance. 

Finally, the author combined regression analysis and discriminant analysis to 
form a classification model which establishes group membership and indicates the 
probable degree of individual success in such a group.' Decision functions match 
individuals to groups. When training requirements for groups such as "fighter 
pilot" or "bomber pilot" or "navigator" are identified (which Cline does not do), 
then tailored learning programs for individuals within those groups can be 
developed, 

DATA SUMMARY: 

Media: Computer-based 

Application: Individualized Instruction 

Evaluation: None 

Environment: Military (Air Force) 

Key Number: 1.2 Select Tasks/Functions 

Authoring Aid Type: Procedure 

Number of Pages: 211 

Number of References: 
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REFERENCE NUMBER: 1.2.2 SELECT TASKS/FUNCTIONS 

CITATION: 

McKnight, A. J., & Adams, B. B. Driver education task analysis. Volume II: Task 
analysis methods. Fin al report (HumRRQ Tech. Rep. 7P-1.'^). Alexandria, Va.: 
Human Resources Research Organization, November 1970. (ERIC Document Reproduc- 
tion Service No. ED 075 624) 

■ ^y, ... 

ABSTRACT: 

During the development of performance-oriented driver education objectives and 
a set of measuring devices for evaluating attainment of the objectives, over one 
thousand behavioral ly-relevant system characteristics were identified. The behav- 
iors identified during the analysis of driving tasks varied considerably in their : 
criticality to the safety and efficiency with which the highway traffic system 
operates. Since an efficient program of driver education must account for this 
variation, an evaluation of behavior criticality became a part of the analysis. 
The approach to criticality evaluation was, of necessity, a judgemental one; there 
were insufficient data to support any empirical determination of criticality. A 
group of authorities in the area of traffic safety— driver educators, enforcement : 
officers, license officials, and fleet safety personnel -- performed the evaluation. 
The plan called for each evaluator to rank, in terms of criticality, three groups 
of 25 behaviors from a total list of 1500 behaviors. This allowed each behavior to 
be ranked five times. However, since seven of the evaluators did not return com- 
pleted evaluations, the number of rankings in some cases was four instead of five. 
The rankings were then transformed to standard scores with a mean of zero and a 
standard deviation of 10. The scores for each behavior were averaged to obtain a 
criticality index for each behavior. An analysis of variance showed a high degree 
of agreement among the evaluators. 

There were four steps in the criticality evaluation: (a) obtaining criticality 
data; (b) selecting an evaluation method; (c) administering the evaluation } and 
. (d) generating criticality indices. 

Data were obtained from a literature review and from analysis of 1100 actual 
automobile accidents. An evaluation method was chosen after a pilot study of three 
approaches — rating, task group ranking, and random group ranking — found that ; 

. B-17 

61 



BIBLIOGRAPHY VOLUME I 1 ? FEBRUARY 1977 

random group ranking provided the only straightforward means of comparing behaviors 
across tasks. Once the data had been obtained and the evaluation method had been 
chosen, the actual administration of the evaluation materials began. One hundred 
evaluators chosen with the aid of the National Highway Safety Bureau were mailed a 
sample of 25 behaviors, the task descriptions, and a set of directions. Upon com- 
pleting a ranking, the evaluator was asked to place a colored card in the stack of 
ranked behaviors beneath the last behavior he felt was critical enough to be re- 
quired of qualified drivers. The rankings were then returned to the authors. 

The data from 94 evaluators were used to generate criticality indices. In 
analyzing the results of the rankings, individual ranks were converted to their 
normalized equivalents. The normalized scores ranged from +2.0 for a rank of "1" 
to -2.0 for a rank of "25," and the mean of zero corresponded to a rank of 13. The 
means of these normalized values and their associated standard deviation were then 
calculated for each; of the 1500 behaviors. 

Since the behaviors were not a representative sample of the population, their 
relative standing to one another would not approximate their standing within the 
population, and so simply ranking the behaviors against one another was not appro- 
priate. Thus, an additional procedure, described below, was needed to create valid 
criticality indices. 

First, a group of the previously ranked behaviors was selected to provide a 
behavior whose mean rank corresponded to each of the 25 ranks. Next, an additional 
group of five evaluators took each of the unranked behaviors and assigned it the 
rank of the behavior in the list of the ranked 25 behaviors to which it most closely 
approximated in criticality. The behavior was then given the same rank value as the 
corresponding ranked behavior. The evaluator was also asked to decide whether the 
behavior was sufficiently critical to be considered a minimum requirement for a 
qualified driver. The results were then processed in the same manner as described 
for the previously ranked behaviors. 

Although the criticality indices remain ultimately subjective and one's confi- 
dence in their validity rests with his confidence in the evaluators, a high level 
of agreement supports confidence while non-agreement destroys it. An analysis of 
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variance indicated a high level of agreement among evaluators concerning the criti- 
cal behaviors. Further, an analysis of variance of inter-task differences indicated 
much similarity in the criticality of behaviors within a particular task. The re- 
sults of each analysis of variance are given in the text of the report. 

f 

Although the procedure suggested by McKnight and Adams is rather involved, 
especially where large numbers of behaviors are considered, it provides a solution 
to the course developer's question of, "What tasks are sufficiently critical that 
they must be taught?", and thereby aids the selection of appropriate tasks and 
functions. 

Four appendices are given: A. Behaviorally Relevant System Characteristics; 
B. Directions to Evaluators of Task Criticality; C. Task Criticality Means and 
Standard Deviations; and, D. List of Evaluators. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key Number: 
Authoring Aid Type: 
Number of Pages: 
Number of References: 



None 

Individualized Instruction 
None 

Civilian 

1.2 Select Tasks/Functions 

Procedure 

45 

None 
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REFERENCE NUMBER: 1.2.3 SELECT TASKS/FUNCTIONS 

CITATION: 

: Powers, L. The systems approach to functional iob analysis: Task analysis nf thP 
physician' s assistant. Volume 1: Task analysis methodology and techniques . 
Winston-Salem, North Carolina: Wake Forest University, Bowman Gray School of 
Medicine, 1971. (ERIC Document Reproduction Service No. ED 059 378) 

ABSTRACT: 

The basic structure of a task statement has two parts: the end result; and the 
worker activity required to bring about the end result. The end results are those 
steps which must be carried out to accomplish the organization's objectives. For 
physician's assistant, typical end results might be "progress notes on hospital, 
home, and office patients" and "hospital discharge summaries for physician's review 
and signature." Typical work activities for the same profession include "perform 
and/or schedule" and "assist the physician with...." These are basic structures 
which every competent task statement should contain. 

Powers, however, has added a phrase to the task statements collected in this 
volume by which task statements may aid in selection of tasks and functions for 
training. The phrase is: in order to . The phrase may be used to indicate an in- 

^ ^^^^y according to required levels 

of competence. Those students with nearly adequate competence to do the intended 
task should receive less training than those who lack a basic competence. Consider, 
i for example, two tasks in electronics training which have different outcomes but a 
sanie worker activity. The tasks are "troubleshooting an oscilloscope" and "using 
an oscilloscope,' and the worker activity is ''turn on an oscilloscope." If a stu- 
dent,::turns on an oscilloscope in order to troubieshbpt it rather than in order to 
operate it, a very different level of training - both in content and extent - is 
required for the end result. Also, students may be expected to bring different 
. prior abilities to each task. 

The collection of task statements for physician's assistant was developed dur- 
ing a systematic sampling of the professional activities of pediatricians, family 
practitioners, surgeons, obstetricians, and internists. Course outlines, training 
objectives, and goals for training physician's assistants were also developed. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key Number: 
Authoring Aid Type: 
Number of Pages: 
Number of References: 



None 

Individualized Instruction 
None 

Civilian 

1.2 Select Tasks/Functions 

Procedure 

130 

None 
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REFERENCE NUMBER: 1.3.1 ^ CONSTRUCT JOB PERFORMANCE MEASURES 

CITATION: 

Gael, S. Employment test validation studies. JSAS Catalog of Selected Documents 
in Psychology . 1974, 4, 95 (Ms. No. 711) 

ABSTRACT: 

Gael reports a study of five Bell System jobs in order to identify job activi- 
ties and dimensions, job products and factors for product quality, relative impor- 
tance of various products and job activities, and the skills required to perform 
the work. The activity analysis results — "activity analysis" being a Bell System 
word substituted for "task analysis" - were translated into employment, tests and job 
performance measures. Gael's discussion is confined to the selection and design of 
the Instruments used to gauge aptitudes and proficiencies. It does not deal with 
results. 

Five Bell System jobs were studied: "(1) Service Representative; (2) Toll Opera- 
tori (3) Directory Assistance Operator; (4) Traffic Service Position; and (5) 
Clerical. As an example of 'thre thoroughness and adaptability of the Bell System 
studies, the Service Representative (SR) and the Clerical Tests will be discussed. 
SR work positions consist of a special desk in which 2500 to 4000 customer recxsrds 
are filed; a position identification board that identifies, by telephone number, 
all the records filed in the desk; a telephone; several reference books; an elabor- 
ate in-out basket for work status documentation; and other incoming and outgoing 
paperwork. SR activities consist of customer contact, clerical, computational, 
and filing efforts which are performed under a steady influx of customer and intra- 
company calls. A call frequency analysis, however, shows that the four most fre- 
quent contacts require: (a) taking orders for newtelephone service or changes to 
existing service; (b) explaining and adjusting telephone bills; (c) recording equip- 
ment malfunctions or service problems and reporting them to maintenance; and (d) 
notifying customers of overdue accounts and arranging payment schedules. 

Two SR job performance measures were developed to measure acquisition and 
application of job knowledge and skills, and to measure job proficiency in a 
standardized situation. The primary purpose of the written Job Knowledge Review 
(JKR) was to determine the comprehension level of company policies, and job 
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procedures and practices. Using multiple-choice and constructed-response questions, 
the JKR covered topics such as SR reference materials, types of equipment available 
to customers, charges for individual pieces or combinations of equipment and ser- 
vices, customer complaints, and intercompany and intracompany communications. The 
JKR was administered to a class of SRs at the end of formal training and required 
about one hour to complete. On the other hand, the Job Performance Review (JPR) 
was a job proficiency measure involving simulation. One administrator initiated 
calls to the SR in a time- and-setting-controlled situation while another specially- 
trained administrator listened to each "customer" contact and evaluated the SR's oral 
performance. The JPR consisted of contact-generated and other clerical work and re- 
^ quired 75 to 90 minutes to complete. Three JPR scores were given for each simula- 
tion: (a) Record Preparation (RP), the sum of the points accorded each part of a 
record prepared or completed as determined by comparison with a model set of re- 
cords with points already assigned; (b) Verbal Contact (VC), the sum of points 
assigned each part of the verbal interaction with the "customer;" and (c) Filing (F), 
the number of records not in the designated location when the JPR was terminated. 

The JKR and the three JPR scores were combined in a composite score that repre- 
sented the acquisition of knowledge and the ability to apply that knowledge on the 
job. In response to management emphasis on customer satisfaction, the Verbal Con- 
tact (VC) score was weighted twice that of the other components. 

The paradigm described for the SR test was followed for the three Operator test 
studies with only minor modifications. 

Nearly 120,000 telephone company employees work at 400 jobs classified as 
clerical. The main problem in developing a job performance measure for clerical 
workers, then, was to reduce systematically clerical jobs to activities coiiinon to 
most clerical jobs. Descriptions of 400 jobs were reduced to eight dimensions and 
work samples in each dimension were obtained which, after refinement and printing, 
resulted in a Clerical Performance Measurement (CPM) package and an administrative 
manual used to train CPM administrators in giving and scoring the CPM. 

The administration, media, and format of the CPM could perhaps be readily 
adapted to a clerically-oriented Defense Occupational Specialty (DOS) such as the 
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Air Force's Inventory Management career field.. below the seven-skill i level . 
Administration of the CPM requires about two days and can be given to groups of 
10 people by one administrator. In addition to the CPM materials and administra- 
tive manuals, other requirements include tape cassettes,' playback units, office 
supplies, a timer, and an enclosed, quiet room that can hold up to 10 study partici- 
pants at separate tables. Completed CPM packages are scored by the administrator 
and independently by an assistant. 

The eight CRM packages consist of five timed clerical activities, three self- 
paced programmed texts, and an audio tape. Each unit is contained in a separate 
folder placed inside an expansion folder. The timed clerical activities, completed 
in accordance with specific instructions, are Filing, Classifying, Posting. Checking 
and Coding. The self-paced units contain programmed texts and examinations that are 
parts of the telephone company training course for clerks. The examinations are in- 
tended to measure the ability to learn the specific content and detail of clerical 
jobs. They are entitled Toll Fundamentals, Punched Card Fundamentals, and Plant 
Repair Service. Job proficiency measures are obtained by combining the eight CPM 
scores for each study participant. 

The five employment test validation studies discussed above covered jobs in 
which about 300.000 Bell System employees work. and. according to Gael, many more 
than that are screened yearly for those jobs. 

DATA SUMf^RY: 

Simulation, programmed text, and audio tape 
Application: Self-paced job performance measurement 

Evaluation: None 

Environment: Civilian 

Key Number: 1.3 Construct Job Performance Measures 

Authoring Aid Type: Procedure 

Number of Pages 8 

Number of References: 5 
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REFERENCE NUMBER: 1.4.1 ANALYZE EXISTING COURSE 

CITATIONS: 

Inventory manageme nt specialist and inventory management supervisor. Specialty 
training standard 645X0 . Washington, D.C.: Department of the Air Force, 
Headquarters, 11 August 1975. 

Materiel f acilities specialist. Plan of instruction 3ABR64730-1 . Lowry Air Force 
Base, Colorado: Lowry Technical Training Center, School of Applied Aerospace 
Sciences, Department of Logistics Training, 2 January 1974. 



id 
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Materiel f acilities specialist. Course chart 3ABR64730-1 . Lowry Air Force Base, 
Colorado: Lowry Technical Training Center, School of Applied Aerospace 
Sciences, Department of Logistics Training, 12 September 1973. 

ABSTRACT: 

Analysis of an existing Air Force course appears in the Specialty Training 
Standard (STS), Plan of Instruction (POI), and the Course Chart. These are 
official. Air Force-prepared documents intended to provide guidance to course 
administrators, instructors, and developers. 

The STS describes in terms of a number-letter code the knowledges and skills an 
airman in a particular career field (such as Inventory Management) musjb have by end 
of training. The number codes indicate required task performance levels, and range 
from 1 (least) to 4 (complete task proficiency). The letter codes are both lower 
and upper case, and indicate task or subject knowledge levels, respectively. Lower- 
case letter codes range from "a" (knowledge of nomenclature) to "d" (knowledge of 
theory). Upper-case letter codes range from "A" (knowledge of facts) to "D" (knowl- 
edge of evaluation). The requirements for task knowledges and skills are given by 
skill level. For example, one task in the Inventory Management career field is 
"conducting inventories", which three-level airmen are required to know to the "A" 
STS level and five- level airmen are required to know to the "B" STS level. That 
means a three-level airmen "can identify basic facts and terms about the subject" 
while five-level airmen "can explain relationship of basic facts and state general 
principles about the subject." The STS is useful in determining a number of task- 
related issues such as selection of training tasks or writing behavioral objectives 
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because it is an officially accurate analysis of required knowledges and skills in 
a particular field. 

The Plan of Instruction (POI) is a publication that identifies units of instruc 
tion for a particular course and indicates the following for each unit of instruc- 
tion: (a) duration of instruction and the day on which it is presented; (b) the 
criterion levels to be reached; (c) support materials; and, (d) the STS reference 
for the course content being presented. 

The Course Chart divides the course into blocks of instruction and specifies 
the sequence in which the lessens are presented. Major equipment items used in a 
course are identified and the number of hours of technical and related training is 
specified. Related training includes commander's calls and briefings, traffic 
safety, and human relations. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 

Environment: 
Key Number: 
Authoring Aid Type: 
Number of Pages: 
Number of References; 



Print 

Conventional and self-paced instruction 

Quality control' by experienced personnel in the career 
field 

Military (Air Force) 

1.4 Analyze Existing Course 

Tool 

Variable by career field 
Variable by career field 



■ •• 70 

B-26 



• AUTHORING AID MDC £1570 

BIBLIOGRAPHY VOLUME I 11 FEBRUARY 1977 

REFERENCE NUMBER: 1.4.2 

CITATION: 

Lewis, W. E., Willow, J. D., Brock, G. R., Lonigro, J. K. , Jr., Eschenbrenner, A. J. 
& Hanson, A. L. Precision tp*>asurinq equipment (PME) individualized instruction 
(Report No. AFHRL-TR-74-46). Lowry Air Force Base, Colorado; Air Force Human • 
Resources Laboratory, Technical Training Division, August 1974. 

ABSTRACT: 

A portion of an Air Force course in Precision Measuring Equipment (PME) 
(3ABR32430-2) was converted from conventional instruction to sel f-paced. individ- 
ualized instruction in order to determine the feasibility of individualizing the 
entire course as part of the Advanced Instructional System (AIS). Self-paced pro- 
grammed texts and audio-visual materials were developed, administered, and evaluated 
for a 90-hour block of instruction. Training time was reduced by 31% with all 
trainees achieving criterion objectives. 

One phase of the study concerned analysis of the existing course, which was 
done in five tasks. The first task was to select course segments for individuali- 
zation according to five selection criteria. Chosen course segments should: 

(a) sample the fuU range of skills and knowledges required in the PME course; 

(b) sample the full range of media requirements in the PME course; (c) contain 
skills and knowledges which are prerequisites for accomplishing frequently occur- 
ring job tasks; (d) contain skills and knowledges essential to both current and 
projected frequently occurring job tasks; and, (e) be able to be implemented with- 
out disrupting normal trainee flow through non-selected segments. 

The second task was to analyze course and job documentation. The following 
doucments were analyzed: (2) Course Chart; (b) Specialty Training Standard (STS) ; 

(c) Plan of Instruction (POI); (d) student study guides, workbooks, programmed 
texts, and handouts; and, (e) the Occupational Survey Report (GSR). The Course 
Chart, POI, and STS proved useful in providing an overview of the course (see 
annotations for these documents in this section, 1.4.1) while the OSR was used to 
obtain data on task, frequency of performance, and total time spent doing the task " 
on the job. 
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The third task was to choose the target population. Several factors were con- 
sidered: (a) size and rate of increase or decrease of target population; (b) loca- 
tion and size of instructional "units"; (c) Airman Qualification Examination (AQE) 
scores relevant to the general and specific knowledges and skills required by the 
PME course; (d) scores on standardized national achievement tests and/or on vali- 
dated local achievement tests; (e) entry skills and learning rates; and, (f) appro- 
priate instructional strategies. 

The fourth task was to examine the training environment. The PME laboratory 
contained 12 classrooms. Those classrooms devoted to academic topics (such as 
basic electronic theory) contained tables and chairs while classrooms given to 
"hands-on" practical performance contained not only tables and chairs but also, 
workbenches and electronic equipment. The primary pieces of equipment in the PME 
course were calibration standards and applied measurement trainers. Since the 
standards and trainers were available in limited quantities, trainees in some 
segments of instructions worked in pairs. 

The fifth task was to select representative segments of the PME course for 
individualization. Air Training Command specified that Block X, Waveform Analysis , 
be individualized, which satisfied selection criteria two through five. Anh-s^ddi- 
tional 24 hours of instruction were selected from other blocks in order to satisfy 
selection criterion 1. 

Eight appendices are included in the report: A. Sample Task Analysis; B. Sam- 
ple Learning Objective; C. Lesson Flowchart; D. Trainee Progress Flowchart; 
E. Instructional Method Questionnaire; F. Block X, Small-Group Tryout Results; 
G. Block X, Large-Group Validation Results; and, H. Trainee Background Data. 

DATA SUMMARY: 
Media: 

Application: 
Eval uation: 
Environment: 
Key Number: 
Authoring Aid Type: 



Programmed texts and audio-visual 
Self-paced,, individualized instruction 
Formative and summative evaluation 
Military (Air Force) 
1.4 Analyze Existing Course 
Procedure ^ ^ 
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Number of Pages: 62 
Number of References: None 

1 
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REFERENCE NUMBER: 1 .5.1 SELECT INSTRUCTIONAL SETTING 

CITATION: 

D. p. Instructio nal hardware/A g uide to arrhi rprfnra l requirerr.gp ts . 
New York: Educational Facilities Laboratories, 1970. (ERIC Document 
Reproduction Service No. ED 043 959) 

ABSTRACT: 

A dated but comprehensive discussion of audio and visual display equipment, 
television-based teaching systems, computer assisted instruction (CAI), and 
student response systems useful in a variety of instructional settings is presented. 
No evaluation of specific systems is given. No arguments are made for which 
systems should be installed, how large the installations should be, or how they 
might be used. The report is intended to "famil iarize the educational conmunity 
with the general nature of various electronic teaching systems." Chapter sections 
begin with a general discussion, and then specific equipment types are considered 
Approximate costs and component flowcharts are frequently presented for system 
layouts. 

The audio equipment chapter considers: (a) tape recorders and phonographs j 
(b) wireless teaching systems emphasizing radio; (c) language laboratories j 
(d) telephone-based teaching systems; (e) dial access systems; and (f) cable 
distribution systems. 



Sections in the visual display equipment chapter include: (a) slide and 
filmstrip projection; (b) overhead projection; (c) opaque projection; (d), motion 
picture projection; (e) film loops including single concept, repeating, and 
cartridge films; (f) microfilm, (g) room darkening and ambient light standards; 
(h) projection screens; (i) lectern control panels; (j) individual student 
carrels; and (j) teaching machines. 

In the chapter on television-based teaching systems, instructionartelevision 
(ITV) is distinguished from educational television (ETV). ITV is defined as tele 
vision programs planned for particular school systems and broadcast via closed 
circuit transmission to specific areas or classrooms. ETV is defined as general 
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educational and informational television programs usually broadcast over public 
television channels. The chapter includes sections on: (a) television transmission 
and distribution systems; (b) production equipment} (c) production space require- 
ments; (d) service requirements for TV production, including air conditioning, power 
and lighting requirements 5 (e) display methods for TV programs; (f) approximate 
equipment costs for open and closed circuit systems; and (g) investment cost 
(equipment plus maintenance) for 150-classroom and 300-classroom systems. 

The CAI chapter discusses: (a) implications of educational computer systems; 
(b) processing requirements; (c) terminal requirements; (d) system options; 
(e) space requirements; (f) environmental requirements; (g) service requirements, 
including methods of cabling and wiring for computer systems; (h) fire protection; 
and (i) approximate costs of a large municipal CAI system. Due to constant 
development of computer-based education equipment, costs cited by the author are, 
perhaps, now unrealistic. But the general information in each section, the learning 
center diagrams of typical CAI learning centers, and the wiring/cabling illustra- 
tions remain informative. 

Student response systems use some type of responder unit at a student station 
and an instructor console that registers responses. The chapter contains two 
sections: (a) a brief description of responder units and instructor consoles; 
and (b) a discussion of three commercial systems. 

Instructional technology terms are defined in Appendix 1, a glossary of 196 
definitions. Other appendices provide the names and addresses of major manu- 
facturers of electronic teaching equipment, a bibliography of suggested readings, 
and data sheets for recording equipment characteristics. 

DATA SUMMARY: 

^6^1*3= Audio-visual, television, computer-assisted 

instruction 
Application; None 
Environment: None 

Key number: 1.5 Select Instructional Setting 
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Authoring aid type: Methodology 
Number of pages: 124+ 
Number of references: 15 
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REFERENCE NUMBER: 2.1.1 



DEVELOP OBJECTIVES 



CITATION: 



Barton, G. E. Performance objectives: A self-instructional booklet . Provo, 
Utah: Brigham Young University Press, 1973. 



ABSTRACT: 



This booklet teaches how to- recognize and write technically correct performance 
objectives. Validated over a two-year period by hundreds of students and several 
instructional psychologists, the booklet is arranged in programmed- text format 
and consists of two parts. 

Part One covers the three aspects of a technically-correct performance 
objective: behavior; criteria; and conditions. With clear examples. Barton 
notes that: (a) specified behavior should be observable and indicate proficiency 
, at a task; (b) specified criteria should be measurable and appropriate, i.e., 
should match or be important qualifications of the desired behavior; and 
(c) specified conditions should describe the exact task characteristics to be 
mastered and list the resource materials available for use. 

A self-check is given at the end of each section. The last self-check serves 
as a post-test for the program and can be t'aken at any time. 

Part Two of the booklet consists of pages for Wfi^. . ..chnically corr^^^^^ 
performance objectives, and a key for critiquing the written performance 
objectives. 

The author admits that writing instructional objectives is not always 
appropriate and includes an article discussing situations where more general 
learning statements — called "worthwhile activities" — should be used. 

DATA SUMMARY: 

Media: Programmed text 

Application: ~ Self-paced instruction 

Evaluation: "Hundreds of students and several instructional 



psychologists over a two-year period 
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Environment: Civilian 

Key number: 2.1 Develop Objectives 

Authoring and type: Tool 

Number of pages: 36 

Number of reference: None 
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REFERENCE NUMBER: 2. 1.2 DEVELOP OBJECTIVES 

CITATION: 

Behavioral objectives training packag e. Amherst, Mass.: Evaluation Service 

Center for Occupational Education, 1971. (ERIC Document Reproduction Service 
No. ED 060 218) 

ABSTRACT: 

The purposes of this training package are: (a) to servje as a user's guide to 
writing and implementing effective behavioral objectives? (b) to provide an 
overview of the services provided by the Evaluation- Service Center for Occupational 
Education (ESCOE); and (c) to suggest an alternative technique to aid the writing of 
behavioral objectives. ( 



f 

V / 



Part I defines behavioral objectives in terms of measurable performance, 
conditions under which the performance occurs, and extent to which the performance 
can be measured. These tenns are analogous to the commonly-accepted teniis of the 
basic structure of a behavioral objective: behavior; condition: and standard. 

Part II discusses the.operation of ESCOE. Essentially, the service is an 
information feedback/evaluation system for a decentralized, voluntary network of 
occupational schools called Local Education Agencies (LEAS). The LEAS submit raw 
behavioral objectives to ESCOE. which are then coded and placed in a computerized 
data bank. When a sufficient quantity of raw (original) objectives are available 
within a given subject area, they are retrieved from a data bank and synthesized by 
a panel of instructors active in that particular subject. The objective synthesis 
process combines raw objectives having the same or similar performance into one . 
larger objective so that all variations of conditions and extent indicated by 
different LEAs are included. Criterion tests are constructed from these synthesized 
objectives which are then made available to the LEAs. Participating LEAs can 
access the total dati^bank, but information identifying the source of a given 
objective, is blanked out when a printout is sent to an LEA other than the one 
submitting the original data. 
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A description of each section and instructions on how to fill in the required 
information are.glyen for the objective input coding form (Behavioral Objective 
Reporting Form #12). 

« 

Part III of the report presents an article entitled "The Operationalizing of 
Fuzzy Concepts." which offers an alternative way of systematically breaking down 
a generalization into its measurable parts, i.e.. into the structure suitable for 
an objective. 

Sample transparencies are available from ESCOE for participating LEAs. A 
gVossary of terms and phrases is given. 



DATA SUMMARY: 
Media: 

•Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Computer-based, print, visual 

Conventional and individualized instruction 

None 

Civil i an 

2.1 Develop Objectives 

Tool 

128 

31 
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DEVELOP OBJECTIVES 



ERIC 



Dinman, F. E., Jr. Instructional objectives: Specificity and behavior . 
Menlo Park, California: Dillman Associates, 1971. (ERIC Document 
Reproduction Source No. ED 061 704) 

ABSTRACT: 

The basic structure of a behavioral /performance objective contains a required 
behavior, a specified standard, and a given condition. However, there are good 
objectives and there are not-so-good objectives. To write good, effective 
objectives, Dillman suggests that several conceptual factors be considered. His 
discussion is presented in prog rammed- text format with a self-review and answers 
• and explanations at the end of each section. 

The first factor is to ensure that an instructional objective rather than an 
educational goal is written. "An educational goal is a long-range and broader 
accomplishment to be achieved in a semester, in a school year, or even during a 
longer school experience, such as upon- completion of high school. An instructional 
objective is, by contrast, a short-range accomplishment to be achieved in a day, 
a week - a relatively shorter time." 

♦ - ' 

The second factor is to prevent an objective from being written so narrowly 
and restrictively that it becomes merely a learning activity or teaching step. 
But "if you are writing objectives that are too narrow, it doesn't necessarily 
mean that you should 'throw them out' — vhen you get your objectives properly 
composed, the 'too narrow' objectives may still be used as teaching points or 
learning activities for the objectives you develop." In other words, objectives 
by nature have a requisite specificity. 

The third factor is to write both the content and the language of the 
objective to the level of the target population, for significant behavior at one 
level may be insignificant at another. "Count to ten in one minute without 
external aids" may be a proper objective for first- graders' but not for calculus 
students ~ and yet the objective contains the basic parts: a behavior, a standard, 
and a condition. g J 
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The fourth factor is to not bury the intent of the objective in the preciseness 
of its words. "The problem in this area arises when a objective states that the 
student be able to cut out the pets in the picture." Does this require skill in 
cutting, or in discriminating? Does it call for indicator behavior or principal 
perfomance? If you want the learner to identify the pets, then demand identifi- 
cation. It is unfair to demand cutting (and the connotation to the pupil that 
this is the required ^kill) when identification is what the objective really 
requires. 

Awareness of these factors plus skill in writing the basic parts of a behavioral 
objective should result in construction of good, effective objectives. 

DATA SUMMARY: 

^^^^ Progratimed text 

Application: Conventional and self-paced Instruction 

Evaluation: None ■ r : . 

Environment: Civilian 

Key number: 2.1 Develop Objectives 

Authoring and type: Tool 

Number of pages: 39 ' 

Number of references: None 
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REFERENCE NUMBER: 2.1.4 DEVELOP OBJECTIVES 

CITATION: 

Objectives for instruc tional programs . Reference wall chart . Huntington Beach, 
California: INSGROUP Inc. , 1975. 

ABSTRACT: 

Objectives for instructional_£ro£ranis is a reusable, multimedia instructional 
package providing up to 25 hours of group or self-paced instruction in writing and 
using measurable objectives. The Reference Wall Chart abstracted here is a four- 
color wall chart containing the major points of the instructional package in a 
concise but easily read format. 

The chart has four divisions. The first division identifies the structure 
of measurable objectives as: (a) an outcome, which is also known as "behavior;" 
(b) an achievement level, also, known as a "criterion;" and (c) the evaluation 
conditions, also known as simple "conditions." An extensive, categorized "shopping 
list" of outcome verbs that can be used to specify achievement levels and evalua- 
tion conditions is given. 

The second division discusses three applications of measurable objectives: 
(a) learning objectives; (b) organizational objectives; and (c) program objectives. 
Definitions and examples for each application are given. 

The third division presents several taxonomies of learning used to analyze sets 
of objectives in-tgrms of logical sequence and complexity levels. Eight taxonomies 
in the cognitive, ''affective, and psychomotor domains are given. A simplified, 
INSGROUP-derived, seven-level classification scheme for seque sets of objec- 
tives on the basis of complexity -is illustrated. 

The fourth division contains brief annotated citations for taxonomies presented 
on the chart, and gives three addresses where collections of objectives may be 
obtained. 
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DATA SUfWARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references; 



Multicolor visual 

Conventional and individualized instruction 
None 

Civilian 

2.1 Develop Objectives 

Tool 

1 

15 
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REFERENCE NUMBER: 2.1.5 
CITATION: 

Preparatio n of learning objectives . Washington, D.C.: Department of the Navy, 
Bureau of Naval Personnel, 1968. (ERIC Document Reproduction Service No. 
ED 058 721 ) 

ABSTRACT: 

Prepared by the Bureau of Naval Personnel , this programmed workbook is intended 
to: (a) introduce the subject of learning objectives and explain their use and 
importance; (b) describe a Navy handbook on writing learning objectives and teaching 
students how to use the handbook as a working reference guide; and (c) provide 
practice in preparing relevant, practical learning objectives. Intended for military 
personnel, the workbook is useful for civilian practitioners too. The workbook 
contains three modules. 

In Module 1, Chapter 1 defines a learning objective as "an instructional goal 
expressed in terms of measurable student performance." Chapter 2 defines terminal 
objectives as representing what the student will be doing on the job while enabling 
objectives should close the gap between what the student knows now and what he is 
to learn. It is noted that an objective is classified as either terminal or enabling 
according to its purpose within the curriculum, and that what represents terminal 
performance for one course may represent enabling performance for a later, more 
advanced course or skill. Chapter! states that learning objectives are important 
because they support "the overall system of Consistent Course Design, Practical 
Instructor Guidance, and Job-Effective Student Instruction." 

Module II introduces the Handbook for Writing Learning Objectives (NAVPERS 
93510-2), the guidebook o^ the Bureau of Naval Personnel for developing learning 
Module II is written in a SCRIPT format (Se It -Con firming Recitation 
in Programmed Teaching) and presents a series of questions about the text. The ~ 
format permits a student to pass quickly over known material, but directs him to 
sections of the Handbook needed for study. Upon completion of Module II, a revi w 
quiz is' given that requires a student to: (a) select the scope of course, topic, 
and lesson objectives; (b) identify each of the three functional elements of a 
learning objective, without error, when presented with a model objective; 
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(c) select examples of knowledge, skill, and attitude categories of learning j 

(d) identify two purposes of a Topic Analysis Worksheet, when presented alternative 
answers in a multiple-choice format; and (e) identify three steps in learning 
objective analysis, when presented with alternative answers in a multiple-choice 
format. 

Module III contains an exercise kit for using the Navy handbook as well as 
the handbook itself. Containing matching sets, constructed-response items, and 
similar learning-by-doing exercises. Module III covers the following topics: 
(a) concept-matching; (b) evaluating learning objectives} (c) constructing learn- 
ing objectives; and (d) a carefully- structured final test in which the student 
writes a knowledge, attitude, and a skill objective. The test is graded upon 
completion. The scoring system consists of stars. For each question answered 
"Yes," 2 stors are awarded; for each question answered "No," no star is awarded. 
A rating scale ranges from 6 stars or less to 10 stars. Ten stars represents 
"outstanding" while 6 stars or less is "unsatisfactory." The scoring system could 
be upgraded. 

DATA SUMMARY: 

Media: Programmed text 

Application: Self-paced instruction 

Evaluation: None 

Environment: Military (Navy) 

Key number: 2.1 Develop Objectives ' 

Authoring aid type: Tool 

Number of pages: 172 

Number of references: None 
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REFERENCE NUMBER: 2.1.6 



DEVELOP OBJECTIVES 



CITATIONS: 



Rose, B. K., Balasek, 'J., Kelleher, J., Lutz, J. L., & Nelken, 1. A progranmed . 
course for the writing of performance objectives. A constructed response 
linear program. Bloomington, Indiana: Phi Delta Kappa, Comnission on 
Educational Planning, 1972. (ERIC Document Reproduction Service No.^' ED 
073 528) 

Rose, B. K., Balasek, J., Kelleher, J., Lutz, J. L., & Nelken, I. Writing 

performance obje ctives. Instructor's manual for teachers and administrators . 
To h? used with a programmed course for writing of performance objectives . 

mington, Indiana: Phi Delta Kappa, Commission on Educational Planning 
>v (ERIC Document Reproduction Service No. ED 073 529) 



Designed to help teachers learn how to write performance objectives, this well- 
writters programmed text covers: (a) identifying goal statements; (b) identifying 
basic parts of a performance objective; (c) writing performance objectives; 
(d) writing cognitive objectives at the levels of knowledge, comprehension, appli- 
cation, analysis, synthesis, and evaluation; (e) recognizing psychomotor objectives 
€.t tl.'e levels of imitation, moniDulation, precision, articulation, and naturaliza- 
tion {that is, achieving autom;j\;-ic and spontaneous responses); (f) recognizing and 
writing perf-niuuice objs^tivoi m the af. -active domain; and (g) recognizing and 
wi-.'ing performance! objecti vc-'- at the pr'ogram level and the instructional level. 

Eawi psrt of the text begins with ob.ie<:t->"ves for the part and continues with 
logical 'y-arranged frames containing frequent embedded questions ari feed back. 

II teachtjs how to classify performance objectives into the cognitive, psycho- 
mOLorv and afiocti /e .'oriidins, and how to specify which level in a domain the 
cbjectivc- is intanded io measure. Part III teaches how to write program level 
Oujfcctives relating to institutional goals and reflecting desired exit levels of 
?,tudent performance. Part III teaches how to write instructional level objectives 
for specific courses, disciplines, or units of work by grade level or curriculum 



ABSTRACT: 



area. 
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A series '^f assit rr t.s intended to review the previous information concludes 
the book. To; cs cov. -id include recognizing complete and incomplete performance 
objectives, completing perfonnance objectives in two of six suggested curriculum 
areas, and identifying simple and complex performance objectives. 

The instructor's manual accompanying the programmed text is designed to enable 
school districts to utilize effective management procedures for inservice training 
of teachers in writing performance objectives. The first four sections of the 
instructor's manual provide management guidelines ?ind instructional aids for 
defining and reviewing the use of programmed materials. Management guidelines 
for de/eloping programmed materials are given in Section 5. The presentation 
includes a description of sequential learning paths as an organizational technique 
for instructional system design. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Progranmed text (Course), Print (Handbook) 

Self-paced instruction (Course and Handbook) 

None (Course and Handbook) 

2.1 Develop Objectives (Course and Handbook) 

Tool (Course and Handbook) 

49 (Course), 38 (Handbook) 

17 (Course), None (Handbook) 
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REFERENCE NUMBER: 2.2.1 



DEVELOP TESTS 



CITATION: 



Brown, L., Brooks, G., Cocks, P. J., and Kersch, M. E. Evaluation for individualized 
instruction. An operational guide for teacher workshops . Downers Grove, 111,: 
Institute for Educational Research, 1 972. (ERIC Document Reproduction Service 
No. ED 066 492) 



Presented in this operational guide for teacher workshops are 11 instructional 
sessions concerned with: (a) educational objectives; (b) global objectives; 
(c) behavioral objectives; (d) hierarchy of learning objectives; (e) purposes of 
achievement testing; (f) techniques of measurement; (g) multiple-choice items; 
(h) basic statistical measures; (i) test analysis interpretation of computer 
output; (j) validity and reliability, and (k) the liaison role of the workshop 
participant. The discussion relating to testing is given below. 

Multiple-choice test ittm^ cor; ist of two parts, a stem or lead and alternatives. 
Faults can be written unintentioria'i iy ir>to either part. A list of 10 common faults 
of multiple-choice test co(V ' . iiction is followed by a fault exercise sheet. The 
faults (a) behavior elicited by test items does not match the behavior stated 

by the ar irsb"..: j^-jective; (b) the item stem does not present a single, definite, 
impor-'.Afki. r; obi em; (c) the item alternatives do not include a single, correct, best 
ansv/er cr "ompletion; (d) an incomplete item stem does not include as much of the 
problem as possible, which may lead to several possible correct alternatives; 
(e) the item alternatives are not stated as briefly as possible; (f) the item is 
not stated in positive terms and, when a negative is used, it is seldom underlined 
or capitalized; (y) the item alternatives are not grammatically consistent with the 
stem; (h) alternatives are not approximately similar in grammatical construction, 
degree of precision, or length; (i) alternatives are implausible and, therefore, 
do not terript students with incomplete or superficial knowleflge of the subject; 
and (j) alternatives contain a "trick" word in an otherwise correct response just 
for the purpose of catching the reader offguard. 



ABSTRACT: 
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In the test analysis instructional session, brief discussions of item dif- 
ficulty, item discriminating power, and correct scores are presented. A series 
of computer printouts of various test analyses follow the discussion, and then a 
list of 52 study questions in four categories is given: (a) format and general 
data; (b) test items; (c) students as individuals and as. groups; a-^d (d) uses of 
data for decision-making. 

Creation of a test-item data bank was one purpose of the workshops. It is 
unknown at this time whether a data bank was in fact created. 



DATA SUMMARY: . 

Media: Computer-based 

Application: Conventional and individualized instruction 

Evaluation: None 

Environment: Civilian 

• Key number: 2.2 Develop Tests 

Authoring aid type: Tool 

Number of pages: 90 



Number of references: 9 
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REFERENCE NUMBER: 2.2.2 DEVELOP TESTS 

CITATION: 

CSE-RBS test evalua tions: Tests of higher-order cognitive, affective, and 
interpersonal skills. Los Angeles: University of California, Graduate 
School of Education, Center for the Sttr!/ of Evaluation, 1972. 

ABSTRACT: 

The Center for the Study of Evaluation (CSE) and Research for Better Schools 
(RBS), Inc. conducted a study to: (a) determine the number and quality of testing 
instruments available for measuring higher-order cognitive, affective, and inter- 
personal skills; and (b) to reveal areas where such instruments were not available. 
Approximately 2600 tests and scales were collected and evaluated. Of 429 categories 
in the three areas, 183 (43%) had no available test, and 179 (42%) contained 10 
or fewer tests. The quality of the instruments was rated predominately poor to 
fair. I 



An evaluation instrument was developed which rated all instj-uments according 
seven criteria: (1) Validity, (2) Examine appropriateness, .e., its suitability 
for an examiner or examinee in terms of comprehension level, physical format and 
response mode; (3) Normed excellence concerning reliability, replicability, and 
refinement of test rssults; ('4) Teaching feedback; (5) Usability,} (6) Retest 
potential, meaning the number of alternate testing forms available ; and (7) Ethical 
propriety, which considered questionable administrative procedures or test content, 
and treatments which were inappropriate, insulting, or offensive !to the examinee. 
As the underlined letters indicate, the evaluation instrument wasj called VENTURE. 

The section for *ests of higher-order cognitive skills included measures of 
disposition, cognitive styles, strategies, and skills. A two-dimensional class- 
ification scheme was developed for the evaluation of cognitive tests. Rows were 
labeled with category descriptors such as "classifying" and "inventing - ideating." 
Columns were headed by content identifiers such as "verbal-semantic," and 
"symbolic." Cells indicated the code assigned to the interaction between the two 
dimensions, and included the number of tests collected and analyzed in each category. 
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The section for tests of affective skills generally included measures of 
traits or dispositions that hold constant ior an individual in different situations. 
As in the previous section, a two-dimensional matrix for test evaluation was 
developed. Rows were identified by six affective skills categories which in some 
cases were subcategorized. Six major affective skills were listed: (1) personal 
temperament; (2) social temperament; (3) attitudes, opinions, beliefs; (4) needs; 
(5) interests; and (6) values. Columns were labeled with classification 
descriptors such as "Reports About Self: Description" and "Behavioral Measures: 
Actual Measures." Cells contained assigned codes and the number of tests collected 
and evaluated for each category or subcategory. 

In the third section, tests of interpersonal skills focusing on specific, 
concrete, social situations and measuring the behaviors exhibited by an individual 
in those situations were evaluated. This aspect generally distinguishes tests of 
interpersonal skills from tests of affective skills. For example, a test which 
measured aspects of a student's relationship with his actual teachers was assigned 
to the interpersonal skills section; but a test which measured the valence of a 
student's attitude toward teachers in general was listed in the section on affec- 
tive skills. A three-dimensional morphological model evaluated test type by role 
by situation. J-^st types were: (a) reports about self; (b) reports about others; 
and (c)»behavioral measures. Roles included relations with: (1) peers; 
(2) authorities; (3) subordinates; and (4) people-in-general. Situations for the 
first roles included: (a) at school; (b) at home; (c) other; and (d) general. 
For the last role, the situation "at work" replaced "other" and "general." Cells 
contained assigned codes for each test category and the. number of tests collected 
and evaluated in that category. 

Two indexes included in the volume provide: (a) the tests and scales arranged 
in alphabetical order with section/cateogry number; and (b) publication, distri- 
bution, and reference sources for the tests and scales. 

DATA SUMMARY: 

Media: Print 

Application: Conventional and indi viduall7:ed ir:" truction 

Evaluation; wnne 
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Environment: Civilian 

Key number: 2.2 Develop Tests 

Authoring and type: Tool 

Number of pages: 322 

Number of references: None 
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REFERENCE NUMBER: 2.2.3 DEVELOP TESTS 

CITATION: 

Gronlund, N. Preparing criterion-referenced tests for classroom instruction . 
New York: Macmillan, 1973. 

ABSTRACT: 

Gronlund presents a practical guide to preparing and using criterion-referenced 
tests (CRT) in classroom instruction. Chapter 1 discusses the nature of six 
requirements for effective CRT. The six requirements are: (a) a clearly defined 
and specified domain of learning tasks; (b) clearly specified standards of per- 
formance; (c) clearly defined behavioral objectives; (d) adequate sampling of 
student performance within each area of performance; (e) test items that reflect 
the behavior specified in the instructional objectives; and (f) criterion scores 
measuring performance to an objective standard. 

Chapters 2 and 3 discuss use of CRT at the mastery level (i.e., achievement of 
minimum standards) and at the developmental level (i.e., achievement beyond minimum 
standards, such as thinking and understanding skills). 

Chapter 4 describes the procedures for planning a test as: (a) delimit the 
unit of instruction to be measured; (b.) state the objectives in specific terms; 
(c) make a content outline; (d) prepare a table of specifications; (e) set pass- 
fail standards; (f) select the f r ^est item to be used; (g) write the test 
items; (h) organize the item pool u test; and (i) create a test-results 
report form for student use. 

Chapter 5 tells how to write short-answer, multiple-choice, true-false, and 
matching test items. It also includes several rules for writing test items. They 
are: (1) ensure that the item elicits the specific behavior called for in the 
objective; (2) make the task to be performed clear and definite; (3) delete non- 
functional material from he item; (4) reject irrelevant factors that prevent the 
student from correct responses; (5) delete irrelevant clues that might lead to 
unintended correct or incorrect responses; (6) v.. ite the test item at the proper 
level of difficulty; (7) insure that the correct answer is agreed upon by subject- 
matter experts; (8) write the item in positive ^orm whenever possible; and (9) write 
enough test items so that a validated test ■<-?m pool may be selected. 
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Chapter 6 discusses the use and appraisal of criterion-referenced tests. 

Three examples of tables of specifications for 40-, 60-, and 100-item tests 
are given which allow a clear breakout between the number of test items per subject 
content area and specific performance type. Appendix A contains a check list for 
evaluating CRTs which may be used after test item construction to ensure that the 
test plan, test items, and test format will adequately measure behavioral outcomes. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Print 

Conventional and individualized instruction 
None 

Civilian 

2.2 Develop Tests 

Tool 

55 

12 
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REFERENCE NUMBER: 2.2.4 DEVELOP TESTS 

CITATION: 

Stevens, 0. C. , and O'Neil, H. F., Jr. Some guidelines for development and review 

gfjiyjtiple-c hoice questions. Austin: University of Texas Press, 1976. 

ABSTRACT: 

This report from Stevens and O'Neil is divided into two sections. The first 
section is a discussion of the advantages of multiple-choice questions, the 
different forms of multiple-choice questions, and methods ^ varying item 
difficulty. The second section lists, with examples, e" -^f^delines for writing 
and reviewing multiple-choice questions. The guidelines as follows. First, 
the multiple-choice item should be meaningful by itself anu should present a 
definite problem. Second, the item stem should. Include as much of the problem 
as possible and should be free from irrelevant materioK Third, negatively-stated 
item stems should be avoided whenever possible. Foi.rl.rs, all alternatives should 
be grammatically consistent with the stem of the item. Fifth, a multiple-choice 
item should contain only one correct or clearly best answer. Sixth, all alterna- 
tives should be plausible; ensure this by drawing alternatives from the same domain 
of Information, by excluding uncorrected facts, and by considering the previous 
learning experience of the student. Seventh, avoid verbal association between the 
stem and the correct answer; an example of verbal association between a word in 
the stem (local ILy) and a word in the alternative (local) is given by the follow- 
ing question: 

"Which of the following agencies should you contact to 
find out about a tornado warning in you locality"? 

a. State farm bureau 

b. Local radio station 

c. U.S. Post Office 

d. U.S. Weather Bureau. 

Eighth, the relative length of the alternatives should not provide a clue to the 



answer. 
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Three additional guidelines are given without examples: (a) seldom use 
special alternatives such as "none of the above" or "all of the above"; (b) place 
the correct answer in each alternative position approximately an equal number of 
times, but in random order; and (c) have competent colleague independently review 
the test items. 

In addition, twelve undesirable multiple-choice item constructions with examples 
and four desirable constructions with examples are given. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Print 

Conventional and individualized instruction 
None 

Military (Air Force) 

2.2 Develop Tests 

Tool 

22 

7 
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REFERENCE NUMBER: 2.2.5 
CITATION: 

Stevens, J. C, and O'Neil, H. F. , Jr. Suggestions for development of test items . 
Austin: University of Texas Press, 1976. 

ABSTRACT: 

An operational definition for test item development has two components. First, 
test Item development is a set of operations by which a syntactic structure is 
assigned to an instructional sentence. Second, test item development is a set of 
operations performed on that syntax which transforms the relevant segments of the 
instructional sentence into test items. Stevens and O'Neil argue that the opera- 
tional definition of test item development is useful for deriving test items froiTi - 
actual instructional sentences through a technique called item transformation. The 
technique assigns a syntactic structure to an instructional sentence in terms of 
subject, verb, object, and prepositional phases and transforms the sentence compo- 
nents into questions. The technique is illustrated with five different types of 
questions. The first two techniques are suitable for rote memory testigg situations 
the last three are intended for comprehension and higher level testing. 

First, verbatim instruction can be made into yes/no, true/false, short answer, 
and multiple-choice questions. Yes/no questions can be derived by simply taking 
a sentence direclEly from the instructional material, deleting the period, and 
adding a question mark. True/false questions can be made by simply taking a 
sentence from the instructional material and adding the appropriate directions at 
the beginning of the test; since most true/false tests have a large porportion 
of true statements, three ways are suggested for deriving false items. Short- 
answer questions can be created by identifying the desired sentence component 
(a subject phrase, for example) and deleting it for the question. 
questions can be derived by a similar procedure. Base sentences are transformed 
into stems by identifying the segment to be deleted and then deleting it. 
Alternatives are generated by identifying the structure of the correct response 
and taking words, phrases, or sentences with similar structure from the instruc- 
tional materials and using them as distractors. 
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Second, transformed verbatim instruction can be made into questions by using 
the wh-transformation. To accomplish the wh-transformation, a test writer must: 
(a) study the structure of the sentence; (b) select the 'segment to be tested} 
(c) delete the segment; (d) replace the deleted phrase with a wh-word and 
(e) shift the wh-word to the front of the sentence. There are four groups of 
wh-words. First, who and what are used to replace human and nonhuman nouns, 
respectively. Second, which is used to replace noun modifiers. Third, Itow, when , 
where, and wh^ are used to replace the various adverbial categories. Fourth, a 

: form of what do is used when the verb is deleted. Alternatives are generated by 

; the process noted above. 

Third, paraphrase questions can be developed by paraphrasing the base sentence, 
deleting a segment from the paraphrase, or transforming the paraphrase into a 
wh-question. A statement is defined as a paraphrase of another if (a) they have 
no substantive word in common, i.e. nouns, verbs, and modifiers, and (b) they 
are equivalent in meaning. The authors note that not all sentences can be 
paraphrased, especially those using proper nouns and words of a highly specific 
or technical nature. 

Fourth, questions can be formed by substituting particular terms for super- 
. ordinate terms. A term is superordinate if at least some of the members of the 
class can be named without repeating the superordinate term. For example, the 
word 'tool" is a superordinate term because each member of the class has a particu- 
lar name, i.e. pliers, hammers, wrenches, etc. To derive a question formed by 
substituting particular terms for superordinate terms, (a) replace every super- 
ordinate term in the base sentence with a particular item, (b) substitute 
synonyms for every remaining substantive word, (c) delete a segment, or (d) form 
a wh-question. 
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•Fifth, questions can be formed by applying the opposite process, that is, 
by substituting superordinate terms for specific terms. The meaning of the 
sentence is enlarged but not changed completely by the procedure. For example, 
the superordinate term "supplies" was substituted for the particular term "pencils" 
in the following sentences: 

base sentence: "The phrase Base Service Store Item tells you that pencils 

are not subject to normal issue procedures." 
transformed sentence: "The phrase Base Service Store Item tells you that 

certain supplies are not subject to normal issue 

procedures." 

To create questions by substituting superordinate terms for particular terms, 
(a) replace particular terms with superordinate terms, (b) replace all substantive 
words with synonyms, or paraphrase where appropriate, (c) delete a segment, or 
(d) form e wh-question. 

Although some of these transformations may appear quite complex, numerous 
examples are clearly presented which show how each transformation "works" on 
actual instructional materials. 



Print 

Conventional and individualized instruction 
None 

Military (Air Force) 

2.2 Develop Tests 

Tool 

23 
7 
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Media: 

Application-: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



AUTHORING AID 
BIBLIOGRAPHY 

•i^FERENCED NUMBER: 2.2.6 

■ CITATION: 



VOLUME I 



MDCE1570 
11 FEBRUARY 1977 • 



DEVELOP TESTS 



Swezey, R. W., and Perlstein, R. B. Developing criterion-referenced tests 
. (Report No. 287-AR18(2)-IR-0974-RWS) . Arlington. Va.: U.S. Amy Research 
Institute for the Behavioral and Social Sciences, Unit Training, and Educational 
Technology Systems Area, Performance Measurements and Standards Work Unit, 
March 1974. (NTIS Document Reproduction Service No. AD A014 987) 



Though the difference between normative-referenced tests and criterion- 
referenced; tests (CRT) may not always be obvious, Swezey and Pearlstein argue that 
a test is criterion-referenced if: (a) the test items are based upon training 
objectives developed from performance objectives external to training; (b) the 
test itself is primarily used to measure mastery; and (c) the test score is 
based upon absolute standards such as job competence rather than relative standards 
such as class standings. Based on those three criteria, the authors present a 
how-to guide for development and administration of CRT. The guide contains review 
questions and answers for each chapter as well as large fold-out flow charts which 
identify the CRT developmental step to which a chapter refers. The CRT developmental 
process is divided into four steps. 



The first step is to develop adequate objectives. Three levels of objectives 
are defined. Level 1 objectives are prepared according to doctrine and/or 
actual, meaningful units of work which occur in an operational environment. 
Level 2 objectives are modified Level 1 objectives which match training resources 
and safety requirements and, indicate what a student must be able to do at the 
end of training. Level 3 objectives — commonly called Enabling Objectives or 
Learning Elements ~ indicate what a student must know and do as prerequisites 
to attaining Level 2 objectives. 

The second step is to develop a test plan by examining practical constraints, 
deciding the type, number, and fidelity (realism) of the test items, planning 
item sampling or sampling among conditions, and documenting the entire test 
plan. Practical examples and a test plan worksheet are given. 



ABSTRACT: 
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The third and fourth steps are to build an item pool and select the final test 
items for instruction. For the item pool, the authors recommend that twice as many 
items be developed as are specified in the test plan so the pool can be evaluated, 
appropriate items selected, and alternate test forms^deyeloped. 

For selecting final test items, specific guidelines for determining the 
correct tryout sample size (versus total student flow), conducting the tryout, 
analyzing tryout results, and reducing the item pool are given. The PHI coefficient 
was chosen as an item analysis technique, and practical examples thoroughly 
illustrate its use. Two worksheets are given, one for recording feedback from 
tryouts and another for summarizing the results of item pool reviews in terms of 
organizational flow. 

The final sections of the manual. Administering and Scoring CRTs and Assessing 
Reliability and Validity, are essentially dependent upon specific testing environ- 
ments and available statistical resources and, therefore, may not be readily 
transferable as authoring tools or procedures. 

Five appendices are included: (a) Checklist for Constructing CRTs; (b) Checklist 
for Evaluating CRTs; (c) Glossary; (d) Square Root Tables j and (e) Review Questions 
and Answers. 

DATA SUMMARY: 

Media: Print 

^ Application: Conventional and individualized instruction 

Evaluation: None 

Environment: Military (Army) 

Key number: 2.2 Develop Tests 

Authoring aid type: Tool 

Number of pages: 210 

Number of references: None 
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REFERENCE NUMBER 2.3.1 DESCRIBE ENTRY BEHAVIOR 

CITATION: 

Bierbaum, G. A., and Planisek, R. J. An index and procedure for readmitting the 
the academically dismissed student . Kent, Ohio: Kent State University, ' 
1969. (ERIC Document Reproduction Service No. ED 063 555) 

ABSTRACT: 

An index and procedure were developed to establish the probable academic success 
of students seeking readmission after academic dismissal. Covering the period 1959 
to 1969, data were obtained on 1460 students from four college deans. The data 
were analyzed, and an index and procedure based on the minimum grade point average 
(GPA) needed to graduate were created. 

The data indicated the range of dismissals for students returning for one term 
as follows: (a) for freshmen, the range of dismissals was 7956 to 94%; (b) for 
sophomores, 62% to 79%; (c) for juniors, 44% to 56%; and (d) for seniors, 9% to 
40%. The farther along a student was at the time ^of his academic dismissal, the 
better were his chances for graduating after readmission. Readmission data indi- 
cated: (a) 10% to 20% finally graduated; (b) 35% to 50% were dismissed again; and 
(c) 15% to 30% did not return after readmission. 

A chart was constructed with credit hours ranging from 1 to 200 along one 
dimension, and the GPA ranging from 0 to 2.00 in .05 increments along the other 
dimension. Two indices were then retrieved from the cells of the chart. The first 
index computed the minimum required GPA needed each term after readmission for 
.graduation. The GPA multiplied by the credit hours needed to graduate (the total 
was called 'earned quality points') minus the earned quality points at the time of 
dismissal provided an indication of the quality point deficit to be overcome for 
graduation. The second index calculated the difference between the cumulative GPA 
at dismissal and the minimum required GPA needed, to graduate. Thus, the minimum 
GPA to be maintained each semester after readmission was obtained. 

Given the two indexes and the four samples of readmitted students, another 
chart was plotted to establish a criticality index. Successful readmissions were 
plotted as stars , and unsuccessful readmissions were plotted as black dots. A line 
drawn to optimize the discrimination between the stars and the dots indicated that, 
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for the Arts and Science College, a minimum GPA performance of 2.20 was required 
for readmission. 

It is noted that more rigorous statistical approaches could be considered in 
relation to their effectiveness when compared to the ease of using this empirically- 
derived index. Samples plotted over time may suggest trends* such as effects 
attributed to changing academic deans. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key Number: 
Authoring aid type: 
Number of pages: 
Number of references: 



None 

Conventional and individualized instruction 
None 

Civilian 

Describe Entry Behavior 

Procedure 

17 

None 
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REFERENCE NUMBER: 2.3.2 



DESCRIBE ENTRY BEHAVIOR 



CITATION: 



Ecternacht, G. J., Reilly, R. R., and McCaffrey, P. J. Development and validity 
of a vocational and occupational interest Inventory (Report No. AFHRL-TR-73-38). 
Lackland Air Force Base Texas: Air Force Human Resources Laboratory, Personnel 
: Research Division, December 1973. 



An inteii-est inventory designed to measure-the vocational interests of enlisted 
men entering the Air Force was developed. The inventory, called VOICE for Voca- 
tional and Occupational interest C.hoice Examination, was based upon 3,072 surveys 
returned by servicemen with at least six months on-the-job experience in eight 
Air Force career fields and an additional 300 recruits in basic training. The 
purpose of the effort was to develop and validate a prototype interest inventory 
to be used in the Guaranteed Enlistment Program. 

Four hundred occupational interest items, each representing an activity, 
were written after completing a survey of the Educational Testing Service's 
test collection, a review of relevant occupational iinterestinventpry, and an 
examination of Air Force Manual 39-1 (which contains descriptions of all Air 
Force jobs) and the Dictionary of Occupational Titles. The items were grouped 
into four sections: (1) occupations (90 items); (2) work tasks (210 items) j 
(3)' leisure activities (70 items); and (4) desired learning experiences (30 
items). These sections composed the interest inventory. 

Four job satisfaction scales were also developed. They were: (1) Job, 
which measured "intrinsic satisfaction with the actual work activity;" (2) Peer, 
which measured satisfaction with one's co-workers; (3) Supervision, which 
measured satisfication with one's supervisor; and (4) Air Force, which measured 
satisfaction with Air Force working conditions. 

Starting with scales developed on half-samples, an elaborate cross-validation 
technique described in Section IV was used. Briefly summarized, a comparison was 
made of the number of "hits" (i.e. , correctly predicting that an individual was 
a member of a career n'eld) achieved when using the instrument versus the number 
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of hits achieved without It. It was concluded that the Instrument was usefu] 
In measuring vocational Interests. However, test-retest reliability was unsatis- 
factory, and the authors remark, "The actual correlations for each scale have not 
been presented since such a presentation might result In an Inaccurate condemnation 
of scale reliabilities. Estimation of scale reliabilities should be performed in 
the future under rigid conditions so that the estimates may be reported." 

An appendix provides a listing of items in the interest inventory. 

DATA SUMMARY: 

Media: Print 

Application: Conventional and individualized instruction 

Evaluation: Method: Cross-validation and test-retest correlation. 



Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of reference 



Sample size: 1,537 airmen for Sample 1, 1535 airmen 

for Sample 2. 

Military (Air Force) 

2.3 Describe Entry Behavior 

Procedure 



17 
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REFERENCE NUMBER: 2.3.3 



DESCRIBE ENTRY BEHAVIOR 



CITATION: 

Mockovak, W. P. Literacy skills and requirements in Air Force career ladders 
(Report No. AFHRL-TR-74-90) . Lowry Air Force Base, Colorado: Air Force 
Human Resources Laboratory, Technical Training Division, December 1974. 



Fifty-six Air Force career ladders representing 35 career fields were examined 
in order to determine: (a) the reading requirement level (RRL) of each career 
ladder; (b) the reading ability level of personnel in each career field; and 
(c) whether a "literacy gap" existed between the two measures. Subject matter 
specialists in each career field performed a readability analysis of random 
samples of training materials, technical orders, and Career Development Course 
(CDC) materials. Also, scores on Airman Qualification Examinations (AQEs) were 
converted to estimated reading grade levels by a validated conversion procedure. 

It was concluded that: (a) reading skills and requirements varied widely 
among career ladders; (b) in many career fields, materials were not written to 
the reading ability level of the user; and (c) as reading ability level of per- 
sonnel dropped in different career ladders, there was no matching drop in reading 
demands imposed on trainees and job incumbents. 

Reading requirement levels (RRL) varied from a low of 10.6 in the Cook career 
ladder to a high of 14.0 in the Telephone Switching Equipment Repairman ladder. 
Two tables give RRL for the 56 career ladders and for the 35 career fields. 
Due to constraints, it w^is not possible to determine what effect job experience 
had on reducing a job's reading demand or to estimate the relative importance of 
reading in career ladders. 

Reading grade level varied from an average of 9.1 in the Aircraft Pneudraulic 
Repairman specialty to 14.6 in the Prograimiing Specialist ladder. The average 
reading grade level across the Air Force career ladders was 10.8. 

Four decision rules for determining "literacy gaps" are given. They range 
from 50-50 (50% of the personnel can read and understand 50% of the reading 
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material they encounter) to 75-85 (85X of the personnel can read and understand 
75% of the material encountcved). The decision >^ules are applied to ten career 
fields* and It Is was concluded, that reading Improvement programs should be applied 
to career fields with minor "literacy gaps" while, for other career fields, 
rewritten materials should be considered. The author notes that the problem of 
"literacy gaps" Is too complex and has too many Implications for training to be 
discussed In detail In this report. 



An appendix gives the estimate average reading grade level of USAF personnel 
by career ladder. 

DATA SUMMARY: 



Media: 



None 

Conventional and Individualized Instruction 
None 

Military (Air Force) 

2.3 Describe Entry Behavior 

Procedure 

32 

15 



Application: 
Evaluation: 
Environment: 
Key number: 



Authoring aid type: 
Number of pages: 
Number of references: 
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: REFERENCE NUMBER: 2.3.4 ; DESCRIBE ENTRY BEHAVIOR 

CITATION: 

Vitola, B. M., Mullins, C. J., and Brokaw, L. D. Quality of the all -volunteer ' 

Air Force - 1973 (Report No. AFHRL-TR-74-35). Lackland Air Force Base, Texas: / 
Air Force Human Resources Laboratory, Personnel Research Division, April 1974. 
JSAS Catalog of Selected Documents in Psychology, 1974, 4, 153. (Ms. No. 820) 

ABSTRACT: 

Characteristics of aptitude, race, age, education, and geographic area of en- 
listment of all-volunteer force enlistees were studied. The following conclusions 
were reached; (a) average aptitudes of 1973 enlistees in the mechanical and elec- 
tronics areas showed an increase over 1971 and 1972 enlisteesj (b) average aptitudes 
of J973 enlistees in the general and administrative areas showed a decrease when com 
pared to 1971 arid 1972 enl istees 1 (c) an appreciable loss of enlistees occurred 
who had completed post-high school education, which was accompanied by an annual 
downward shift of percentages of first- term airmen scoring in the higher centile 
levels of the Airman Qualification Examination (AQE) ; (d) compared to other areas, 
the 1973 aptitude level patterns of South-Southwest enlistees were not consistently, 
low, as they were in 1970 through 1972; (e) 1973 aptitude levels of black enlistees 
were higher than 1970 through 1972 aptitude levels, across both geographic areas 
of enlistment and total sample; (f) a need may exist to offer incentives to 
high-aptitude people in the general and electronics areas j and (g) Air Force 
operational capability does not seem to be significantly affected by the zero- 
draft environment. The conclusions are based on, data from 324,935 male, nonprior- 
service basic trainees. The data source was the Air Training Command (ATC) per- 
sonnel processing files on nonprior-service accessions. 

Three additional points are made. First, the removal of the draft as a 
motivator for enlistment resulted in a decreased number of accessions completing 
education beyond the high school level. In 1970, 18 percent of the total 
accessions had 13 or more years of education; by 1973, the. number dropped to 
7 percent. Second, the discharge rate for first- term airmen after four years 
was 79 percent. Third, the average difference on mean scores for AQE aptitude 
composites between blacks and nonblacks decreased. 
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DATA Sl!?®1ARY: 

Media: None 

Application: Conventional and individualized instruction 

Evaluation: ^ "None 

Environment: Military (Air Force) 

Key number: 2.3 Describe Entry Behavior 

Authoring and type: Procedure 

Number of pages: 14 



Number of references: 7 
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REFERENCE NUMBER; 2.4.1 DETERMINE SEQUENCIAL STRUCTURE 

CITATIONS: 

Dansereau, D. R., Evans, S. H., Wright, A. D., Long, G. and Atckinson, T. Factors 
related to developing instructional information sequences: Phase I . 
(Report No. AFHRL-TR-73-51 (I) ) . Lowry Air Force Base, Colorado: Air Force 
Human Resources Laboratory, Technical Training Division, March 1974. 

Dansereau, D. R., Evans, S. H. , Atckinson, T. A., and Long, G. L. Factors relating 
to the development of optimal instructional sequences (Report No. AFHRL-TR- 
73-51(11)). Lowry Air Force Base, Colorado: Air Force Human Resources 
Laboratory, Technical Training Division, June 1974. 

"ABSTRACT: 

In Phase I of the study, an extensive review and synthesis of the literature 
relating to the impact of selected variables upon the design and development of 
effective instructional sequences were conducted. A major conclusion drawn was - 
that the findings^ of previous studies were consiste'ntly inconsistent. Some 
studies found greater performance with application of particular sequences while 
others did not. A portion of the inconsistency was attributed to methodological 
faults. For example, a number of studies reported that branching sequences 
(versus fixed sequences) resulted in better terminal performance and/or less time 
to complete instruction. Although branching was the variable being manipulated 
intentionally, it generally led to fewer items being presented in comparison to 
the fixed sequence. Thus, it was possible that the evident improvement was due 
in fact to a reduced number of instructional items rather than branching per se . 
Further, a fixed sequence with the right items dropped may produce thie same results 
as the branching technique. 

Three other factors causing inconsistent findings were: (a) lack of systematic 
procedures for describing information structures and for developing instructional 
sequences; (b) lack of sensitive dependent measures for assessing sequence effects; 
and (c) lack of attention to individual differences in information processing 
styles and capacity. 
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In addition to the literature review and analysis, three pilot studies were 
conducted in preparation for the main experiment reported in Phase II. The pilot 
studies: (1) identified an appropriate presentation rate for the main experiment 
(45 seconds per page) ; (2) selected Air Force materials of differing difficulty 
for the main experiment ; and (3) chose two instructional sequences plus the 
original Air Force sequences for the main experiment. 

The main experiment, reported in Phase II, was designed to: (a) assess the 
utility of multidimensional scaling (called INSCAL) as a procedure for describing 
the information structure of Air Force instructional material and to investigate 
■ INSCAL's potential for generating effective instructional sequences; (b) determine 
if information sequence requirements interacted with a measure of intellectual 
ability, namely, the Delta Reading Vocabulary Test; (c) determine the effect of 
two alternative presentations (verbal and pictorial/graphic) on test performance; 
(d) assess the interaction of instructional material difficulty with other indepen- 
dent variables; and (e) determine the usefulness of two new performance measures, 
a concept-close procedure and a simi larity- judgement methodology. 

Based on data from 180 students, the conclusions were: (a) INSCAL was useful 
for defining information complexity of technical material, developing instructional 
sequences within technical material, and providing an index of expert inter-rater 
consensus; (b) alternative instructional sequences affected student performance; 
(c) small performance differences resulted when pictorial technical information 
sequences were compared to verbal print sequences; (d) increased technical 
difficulty of instructional materials decreased student performance; (e) reading 
aptitudes were significantly related to performance in both pictorial and verbal 
. print presentation modes; and (f) instructional sequences did not interact with 
student aptitudes. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring and type: 
Number of pages: 
Number of references: 



Programmed text and visual 
Individualized instruction 
None 

Military (Air Force) 

2.4 Determine Sequence and Structure 

Procedure 

91 + (Phase I), 37 (Phase II) 
110 (Phase I), 1 (Phase II) 
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REFERENCE NUMBER: 2.4.2 



DETERMINE SEQUENCEMND STRUCTURE 



CITATION: 

Intensification of the learning process: Diagnostic instruments Learner stat e 
. check list evaluation response form . Ooylestown, Pa.: Bucks County Public 
schools, February 1970. (ERIC Docume^^t Reproduction Service No. ED 063 345) 



Intended for elementary public schools but also useful for more advanced grades, 
check lists for determining studentsV positions along a curriculum scale are pre- 
sented. A behavioral objectives evaluation response form is included. The check- 
lists and response form are designed to clarify such questions as: (a) should 
behavioral objectives, and, hence, instructional materials, be grouped by grade or 
across grade level by subject matter; and (b) should behavioral objectives be 
grouped by curriculum area? Clarification results from the clear listing and 
specification of learner states for various curriculum areas, and from the response 
form which describes the extent to which a concept or skill being taught is learned 
and provides a means for recording successive trials or responses until mastery is 
achieved (maximum number of trials in this particular application was 4). 

To determine how well a student achieves behavioral objectives, 'the response 
form uses four numbered codes. The first code is 0 (zero) and indicates that the 
student is utterly unable to perform the behavioral objective performance require- 
ments. Reevaluation of the diagnostic input as, related to the individuaTs 
prescription is needed. The second code is 1 and indicates some measurable ability 
to perform in the manner prescribed but only after additional prompting. Reevalua- 
tion is still necessary. T.he third code is 2, meaning that the learner's opera- 
tional level indicates partial mastery, but requires strengthening in the specific 
context of the prescribed o' ;.»ctive(s). Reevaluation is still necessary. The 
fourth code is 3, indicating that the learner's operational level approaches the 
optimum level . He is able to achieve the performance level required by the 
objective. In fact, such outstanding performance may be achieved that reevaluation 
is needed in order to restructure future objectives. 
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Although the above procedure is a simple approach, it may provide a good 
manual backup for assessing student performance and determining learning sequences 
in computer-based learning systems. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Print 

Conventional and individual ized instruction 
None 

Civilian 

2A Determine Sequence and Structure 

Tool 

71 

None 
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REFERENCE NUMBER: 3.1 SPECIFY LEARNING EVENTS/ACTIVITIES 

No tools or procedures were identified for this activity. 
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3.2 Specify Instructional Management Plan and Delivery System 

This ISD activity consists of three principal steps. First, appropriate media 
for use in the instructional system are selected. A systematic approach to media 
selection requires consideration of the nature of the objectives and of the learning 
category for each group of objectives, the use of learning guidelines and the 
learning activities required by those guidelines. Sinca instructional requirements 
vary for different learning objectives, equipment and faculty constraints, cost of 
procurement and replacement, development costs, and study characteristics must be 
considered. Second, a course management plan is developed. It indicates exactly 
how instruction is to be conducted, how students are to be managed and tested, what 
instructors and support personnel are to do, and how all system elements interrelate. 
Third, a system management plan based on these decisions is prepared which, in 
addition to being a clear explication of system policy and operation, assists in 
internal evaluation of the system. 

More than any other ISD step in the- iPISD model, specifying an instructional 
management plan and delivery system hinges on decisions based upon policy. Virtu- 
ally no decision is made in this activity which does not affect several different 
activities, commit various resources, or affect the utilization of personnel and 
facilities. The decision-maker must be aware of the possible alternatives to a 
particular plan and provide— or be provided with— information about facilities, 
personnel, and other available resources. Decisions made in the execution of this 
ISD activity are based upon something beyond, or in addition to, the evidential 
rightness or wrongness of a technical approach. The decisions are based upon con- 
straints, implications and priorities. In a word, they are based upon policy, 
and policy cannot be procedural i zed in the form of a generalized, transportable 
tool or procedure for course developers. 

For this reason, the references in this section are handled differently from 
•references in other sections. First presented are six abstracts of documents 
which are seen as useful to the decision-making process in any environment or 
instructional setting. Following that is a list of citations only which are 
representative of the type of literature which may aid the decision-making process 
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that is so much a part of specifying an instructional management plan and delivery 
system. Whether or not a particular citation is considered a tool or procedure 
depends upon the context of the instructional or delivery system being considered. 
These citations are offered to provide some indication of the types of activities 
contained in this ISD step and, for those interested in a specific area, to provide 
some leads to additional literature. 
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REFERENCE NUMBER: 3.2.1 



SPECIFY INSTRUCTIONAL MANAGEMENT PLAN 
AND DELIVERY SYSTEM 



CITATION: 



Bretz, R. The selection of appropriate communication media for instruction : 
A guide for designers of Air Force technical training programs (Report 
No. R-601-PR). Santa Monica, Calif.: RAND Corporation, irebruary 1971. 



ABSTRACT: 



Bretz defines communication media as a complete system for the communication 
of messages, which generally encompasses a process for the production of programs, 
a process for the delivery of these programs, and a means for program display. The 
primary input to the system is the message it processes. Secondary inputs to the 
system consist of capital equipment, labor, expendable materials for program 
planning and production, and engineering and administration functions. The primary 
output of the system is the message that was put in a suitable format, i.e., 
displayed programs displayed data. It is but one element of an instructional 
system, since it helps the instructional system to bring about student learning. 
When the primary purpose of a communication medium is to cause learning, then it 
may be considered an instructional medium. This report addresses only "self- 
contained corranunication systems which have reproducable software; supplementary 
media such as-audio and visual aids, which only assist a teacher in presenting a 
lesson, are not considered." 

Communication media, equipment configuration, and program content for eleven 
different uses of communication in instruction are discussed: (a) providing 
knowledge of learning objectives; (b) motivating the learner; (c) presenting 
information; (d) stimulating discussion ;(e) directing learner activities; 
(f) conducting drill and practice; (g) reinforcing learning j (h) providing a 
learner-simulator interface; (i) evaluating learner progress and program effective- 
ness; (j) assisting the administration of instructional systems; and (k) assisting 
the research_^ function of instructional systems. A procedure is 

described for selecting media based on a given instructional need. The selection 
is based on required learning objectives, subject matter content, learner popula- 
tion characteristics, and the instructional method chosen (such as lecture, self- 
paced, or a combination of both)*. Four specific questions in media selection are 
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considered; (a) is there a need for visual presentation; (b) is there a need for 
audio presentation; (c) is there a need for full motion; and (d) what media mix is 
optimum for presenting abstract information? 

A comprehensive glossary of terms used in the report and for media use in 
general is given. 



DATA SUMMARY: 
Media: 

Application: ' 
Evaluation: 
Environment: 
Key number: 

Authoring aid type: 
Number of pages: 
Number of references: 



Audio, visual, video 
Individualized instruction 
None 

Military (Air Force) 

3.2 Specify Instructional Management Plan and Delivery 

System 

Procedure 

65 

10 
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REFERENCE NUMBtK: 3.2.2 



SPECIFY INSTRUCTIONAL MANAGEMENT PLAN 
AND DELIVERY SYSTEM 



CITATION: 



Lonigro, J. K. , & Eschenbrenner, A. J., Jr. A model for selecting the media in 
technical/vocational education. In H. Hitchens (Ed.), Selecting media for 
learning: Readings from Audiovisual Instruction * Washington, D.C.: 
Association for Educatiqnal Communications and Technology, 1974. 



A model is presented for selecting media for technical/vocational education. 
Five factors are considered: (a) types of learning required for a job task; 
(b) proficiency levels required^ (c) types of learning/media matches; (d) profi- 
ciency levels achieved by various learning types/media matches; and (e) relative 
production costs of various media. 



Two applications of the model are discussed ia,±he. area of electronics 
technical training. Three steps are required for applying the model: (a) identify 
the type of behavior to be learned; (b) specify the selection constraints in terms 
of required proficiency and acceptable cost; and (c) plot proficiency which can be 
achieved by specific media versus costs for specific media. 

A "Gameboard for Media Selection" is given. One axis indicates proficiency 
levels from low to high. Another axis indicates' production costs from low to high. 
Data for plotting the gameboard are given in Figures 1 and 2. Figure 1 has eight 
rows for media and five columns for types of learning. The cells of the matrix 
contain the proficiency levels which can be achieved -by using the types of media. 
Ff pre 2^" g^^^^ th'e production costs for ^ight types of media in terms of relative 
lows to highs. When the required proficiency level for a type of learning and the 
acceptable cost for media production are identified, they are plotted on the game- 
board. A clear comparison of type of media versus cost and proficiency level 
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results. 
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DATA SUMMARY: 

Media: Still pictures/graphs, motion pictures, television, 

simulation, audio recordings, programmed instruction, 
tape/slide, computer assisted instruction. 

Application: Media Selection 

Evaluation: None 

Environment: Military (Air Force) 

Key number: 3.2 Specify .Instructional Management Plan and Delivery 

System 

Authoring aid type: Tool 
Number of pages: 4 
Number of references: 6 
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CITATION: 



National Security. Industrial Association. Survey of data-presentation concepts 
and techniques . Washington, D.C.: Author, Technical Publications Panel, 
Logistics Management Advisory Committee, January 1973. - . 

ABSTRACT: 

One way to reduce the growing maintenance expense of increasingly complex 
military systems and equipment is to provide more effective technical documenta- 
tion, i.e., better maintenance procedures and reference manuals, for use by 
technicians with little training and experience. This report, in the format of a 
user's guide, describes new mai ntenance concepts and techniques which. Compared to 
conventional technical manuals, usually results in superior technician performance. 
"In general , this is true for the time required to isolate faultsi the number of 
errors made during maintenance,- and often the length of the technician ';s training U ; 
period." 



The new concepts and techniques are designed to solve two kinds of problems: 
maintenance/troubleshooting, and documentation. The second problem is perhaps of 
more interest to course developers who are concerned with text-handling and instruc- 
tional materials reproduction methods. The report breaks the documentation problem 
into eight areas: (1) providing condensed detail level ; (2) decreasing manual / 
size/bulk; (3) improving access to information; (4) supplying more durable product; 
(5) reducing shipping/storage requirements; (6) facilitating updating; (7) ensuring 
timely availability; and (8) reducing costs of manual production. 

The new concepts and techniques which address the documentation problem are: 

(a) Computer-Aided Trouble Analysis (CATA) developed by Vought Aeronautics Co. 

(b) Malfunction Detection, Analysis and Recording System (MADAR) and its Ground 
Processing System (GPS) developed by Lockheed-Georgia Co.; (c) Condensed Servicing 
Data (CONSD) developed by General Electric Co.; (d) Microfilmed Maintenance Manual 
Data Dissemination (M3DD) developed by the 3M Co.; (e) Automated Data Preparation 
Evaluation Program developed by McDonnell Aircraft Co.; (f) SAFEGUARD Maintenance 
Data System (MDS) developed by Western Electric Co.; (g) Functional Layout and 
Presentation System (FLAPS) developed by Hughes Aircraft Co.; (h) Condensed 
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Maintenance Guides developed by Hughes Aircraft Co.; (i) Spares Test Capability 
Report developed by General Dynamics Corp.; and (j) Computer Graphics and Visual 
Module System developed by William Fetter and currently used by the Boeing Company. 

Details about each new concept and technique were obtained from replies to a 
questionnaire sent to member companies and organizations or the National Security 
Industrial Association (NSIA). Specific;.,.details about each concept and technique 
include: (a) name of concept/technique ; tb) name of developer; (c) description of 
concept- technique; (d) operation/maintenance/manual problems which concept is 
intende . to solve; (e) how the concept solves the operation/maintenance/manual 
problem(s) ; (f) where the concept is being used; (g) test/study results, if any; 
and (h) the person to contact for additional information. 

Since the primary purpose of the report was to collect, organize, and 
disseminate information, no concept or technique was evaluated by the NSIA. 

"The reader should use this report for a brief description of the data 
presentation concept/technique submitted, and contact the contributors and sources 
given in the report body, if additional information is desired." 



DATA SUMMARY: 
Media: - 
Application: 

Evaluation: 
Environment: 
Key number: 

Authoring aid type: 
Number of pages: 
Number of references: 



Print, Computer-based 

Text-handling and instructional materials reproduction 

systems 

None 

Civil ian 

3.2 Specify Instructional Management Plan and Delivery 

System 

Procedure 

153 

None 
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REFERENCE NUMBER: 3.2.4 SPECIFY INSTRUCTIONAL MANAGEMENT PLAN 

AND DELIVERY SYSTEM 

CITATION: 

Pritchard, R. D., Von Bergen, C, Jr., & DeLeo, P. J. Incentive motivation 

techniques evaluation in Air Force technical training (Report No. AFHRL-TR- 
74-24). Lowry Air Force Base, Colorado: Air Force Human Resources Laboratory,' 
Technical Training Division, November 1974. 

ABSTRACT: 

Application of incentive motivation techniques has resulted in improved 
performance in classrooms, mental institutions, and industry. This report 
describes a study conducted at Chanute Air Force Base which evaluated the effec- 
tiveness of incentive motivation in Air Force technical training. The incentive 
motivation program was implemented in two courses. Weather Observer and Aircraft 
Electrical Repairman. The program was evaluated in terms of feasibility, effect 
on training performance, and cost-effectiveness. 

Three incentive systems were implemented sequentially: (a) high feasibility 
incentives awarded on; the basis of performance ; (b) high feasibility incentives 
awarded on the basis of effort; and (c) high and low feasibility incentives awarded^ 
on the basis of effort. The first system's incentives consisted of commendation 
letters, time off, avoidance of work details, choice of uniform, and avoidance of 
marching in formation. The incentives were dispensed through a point system 
whereby a student earned points by improving his exam scores or decreasing his 
time-to-complete. The points could then be spent on the incentives. The second 
system's incentives were similar to those of the first system but points were 
awarded for effort rather than pure performance. A student was assigned a target ■ 
score according to his ability. Achievement above this target score was considered; 
a reflection of effort, and points were awarded accordingly. The third system's 
incentives were financial and consisted of U.S. savings bonds, gift certificates, 
chits for various base facilities, and round trip bus transportation. Students 
earned points by the individual target score method. 

Results of the study indicated that: (a) the third system had the greatest 
positive impact on the time to complete the Aircraft Electrical Repairman course; 
(b) none of the incentive systems improved exam scores; (c) all incentive systems 
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generally decreased remedial instruction probations, and counselling sessions of 
the Weather Observer course; (d) all incentive systems produced increases in 
reenlistment intentions in both courses; (e) the incentive systems had neither a 
positive nor negative effect on job satisfaction or overall attitude toward the 
Air Force; and (f) students slightly favored the incentive system while instructors 
were slightly unfavorable toward it, but neither group found the mechanics of the ' 
systems overly burdensome. 

Conclusions of the study were: (a) incentive systems 1 and 2 were not effec- 
tive because the incentives, while highly feasible to implement, were relatively 
weak in attractiveness; (b) implementation of system 3 resulted in significant 
increases in course performance; (c) cost benefit analysis indicated that the 
savings were more than enough to offset the cost of the incentive systems (d) the 
lack of impact on the Weather Observer course was due to the ceiling effect where 
average performance was already at a very high level; and (e) the effort-based 
incentive system was not understood by students. 

Implications of the study for successful implementation of incentive motivation 
techniques in Air Force technical training were: (a) incentives must be fairly 
powerful; (b) increases in course performance for trainees must not be difficult 
to attain; (c) self-paced courses are most appropriate for incentive techniques; 
(d) more frequent reinforcement should occur; and (e) authority figures and peers 
should provide positive social reinforcement in order to enable the student to 
achieve high performance. 

DATA SUMMARY: 

Media: None 

Application: Student Management 

Evaluation: None 

Environment: Military (Air Force) 

Key number: 3.2 Specify Instructional Management Plan and 

Del ivery System 

Authoring aid type: Procedure 

Number of pages: 294 

Number of references: 173 126 
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REFERENCE NUMBER: 3.2.5 SPECIFY INSTRUCTIONAL MANAGEMENT PLAN 

CITATION: AND DELIVERY SYSTEM 

Richardson, F.C.,0' Neil, H. F., Jr., Grant, R. D., & Judd, W. A. Development and 
prelimina ry evaluation of an automated test anxiety reduction program for a 
computer-b ased learning situation . Austin: University of Texas at Austin, 
Computer-Assisted Instruction Laboratory, January 1973. 

ABSTRACT: 

This report concerning an automated test anxiety reduction program is divided 
into two parts. First, the literature investigating the nature of test anxiety 
and its remediation by behavior therapy techniques is reviewed. Second, the 
development of an automated program for the reduction of test anxiety in a computer- 
based learning situation is described. 

The test anxiety program consists of four components. The first component is 
a 90-page manual written for the program. The manual describes and diagrams the 
process whereby attention is directed to the task (i.e., the test) instead of to 
oneself, provides detailed information about the behavioral and emotional dynamics 
of test anxiety and various strategies for coping with it, .and contains a series 
of written exercises in which the student outlines for himself behavioral strate- 
gies for coping with test anxiety. The second component of the program is a 
videotape presentation of a female student modeling effective and ineffective' 
management of test anxiety. Tha third component of the program consists of a 
modified desensitization procedure of deep muscle relaxation and self-visualization 
of coping with mild anxiety in a graded series of test-taking scenes. The final 
component of the program requires that students practice responding to test-like 
questions at a computer terminal . In addition to the test-like questions, the 
terminal also presents instructions that remind students of the slow, deep breath- 
ing technique for combating physical tension and anxiety, and of the importance of 
paying attention to the test and avoiding self-oriented worry. 

The test anxiety reduction program was applied to 16 undergraduate education 
majors at the University of Texas at Austin taking a required course in Educational 
Psychology during the fall semester of 1972. One part of the course was a sequence d 
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of five computer-managed instructional modules. The results of the program 
indicated: (a) a striking reduction in general test anxiety and state anxiety for 
the treatment as compared with control subjects; (b) no difference between the 
module test scores of the treatment group before and after treatment, and no 
difference between the module test scores of treatment as compared with control 
subjects after the former had completed the test anxiety programj (c) subjects 
expressed generally strong positive feelings about the program; and (d) subjects 
felt that the manual written for the program related much useful information con- 
cerning test anxiety coping strategies. 

It was noted that deep muscle relaxation and visualization of hierarchy scenes 
occasionally (though not in this particular situation) give clients difficulty 
and requires the assistance of a professional. It was recommended that a probabl^^ 
next step would "explore eliminating the relaxation and desensitization component 
of the program in favor of an increased emphasis on practicing test-taking and 
anxiety management ski.lls at the terminal on test-like materials." 
ji 

DATA SUMMARY: 

Media: Print and videotape 

Application: Student management 

Evaluation: None 
Environment: Civilian 

Key number: 3.2 Specify Instructional Management Plan and 

Delivery System 
Number of pages: 41 
Number of references: 74 



128 
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REFERENCE NUMBER; 3.2.6 SPECIFY INSTRUCTIONAL MANAGEMENT PLAN 

AND DELIVERY SYSTEM 

CITATION: 

Siegel, A. I., Federman, P. J., & Sellman, W. S. A survey of student measurement 
and course evaluation procedures within the Air Training Command (Report No. 
AFHRL-TR-74-5). Lowry Air Force Base, Colorado: Air Force Human Resources 
Laboratory, Technical Training Division, July 1974. 

ABSTRACT: 

The reactions of 139 training evaluation, student measurement, and training 
manager/instructor personnel to the Air Force Instructional Systems Development 
(ISD) system are reported. The Air Force ISD system consists of five steps. The 
first step is analysis of system requirements. Tasks performed within the overall 
operational environment are identified. The second step is definition of educa- 
tion or training requirements. The tasks requiring instruction, the {jroficiency 
levels desired, and the required resources are determined. The third step is 
development of objectives and tests. The fourth step is planning, developing, and 
validating instruction. The fifth step is conducting and evaluating instruction. 
The general purpose of the ISD process is to graduate students who meet job perfor- 
mance requirements. 

. Although each group generally regardled the ISD procedures favorably, a number 
bf problems with the use and implementation of ISD were identified. First, there 
was a need for a more general and extensive method for orienting supervisory 
personnel in the attributes and goals of the ISD system. Second, there was a need 
for more personnel who are trained not only in their individual assignments but 
who also possess adequate insight into the specific aspects of other ISD require- 
ments. Third, there was a need to reduce the time required to apply the ISD 
procedures. Fourth, there was a need for streamlining the ISD procedures. Fifth, 
there was a need for evaluation of how well each of the ISD steps is performed. 
And sixth, some users elected to implement ISD through performance of a selected 
subset of the ISD system steps. Since each ISD step is dependent on a prior step, 
such abbreviation leads to faulty implementation. 
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Student measurement techniques and course review techniques were also surveyed. 
The general consensus toward each set of techniques was favorable but, again, some 
problems were identified. Problems with student measurement techniques included: 
(a) some written tests lacked reliability: (b) standards for updating tests were 
inconsistent; (c) parallel test forms may have been required but were not available; 
and (d) use of confidence and hierarchical testing approaches were rare. The prin- 
cipal criticisms of course review techniques (student, instructor and graduate 
critiques) were (a) their lack of objectivity; and (b) the interpretation of data 
by personnel unfamiliar with the total subject matter content or with the context 
in which the recommendations were set. 

Among the recommendations for program improvement are: (a) development of an 
information dissemination program designed to clarify misconceptions about the ISD ' 
system and to orient training personnel in the attributes and goals of the ISD 
program, (b) increased training for personnel involved in ISD use and application, 
(c) development of job aids and work methods, along with an increase in communica- 
"tions efficiency for the purpose of "streamlining" the ISD procedures, (d) evalua- 
tion of the proficiency with which each ISD step is performed, (e) establishment 
of guidelines for updating tests, (f) use of alternative or parallel tests where 
required, (g) determination of the predictive validity of tests used, (h)- allowing 
sufficient testing time for the purposes on hand, and (i) development of course 
review techniques that can yield logitudinal, objective, and quantitative 
information. 

Although these results and conclusions may not apply to other branches of the 
service or to the civilian environment, the methods and procedures of this inter- 
view survey may prove useful to other course developers who are concerned with the 
effectiveness of their particular materials development methodology. 

Appendix A provides details about the interview procedures. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 

Number of pages: 
Number of references: 



None 

Instructional materials and system evaluation techniC|ues 
None 

Military (Air Force) 

3.2 Specify Instructional Management Plan and 

Delivery System 

114 

5 
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ADDITIONAL CITATIONS UNDER REFERENCE NUMBER 3.2: SPECIFY INSTRUCTIONAL MANAGEMENT 

PLAN AND DELIVERY SYSTEM 

Course Management-Students 

Beavers, W. S. Effect of two counselling treatments upon the task-oriented 
behavior of seventh graders . Paper presented at the Personnel and Guidance 
Association Annual Convention, Atlantic City, New Jersey, April 1971. 

McKeen, R. L., & Eisenberg, T. A. On using student-generated sequences in the 
development of a learn ing hierarchy , improving Human Performance. A Research 
Quarterly, 1973, 2, 97-106 

Scheuerer, D. T. The placement of students in viable learning situations through 
the use of ach ievement tests and systems engineering rather than through annual 
promotion and retention. Volume II of II. Final Report. Boulder: University 
of Colorado, Biological Sciences Curriculum Study, June 1970. (ERIC Document 
Reproduction Service No. ED 057 069) 

Stewart, J., & Clark, R. L. University of Maryland student scheduling algorithm. 
Proceedings of the 23rd National Conference of the Association for Computing 
Machinery .' 1968, 555-562 

Timrtireck, E.M. ADVISER - a program which advises students on courses. Proceedings 
of the 23rd National Conference of the Association for Computing Machinery . 
1968, 535-553 

Course Management-Instructors' 

Devaughn, J. E. A manual for developing reasonable, objective, nondiscriminatory 
standards for evaluating teacher performance . State College: Mississippi 
State University, Educational Services Center, September 1971. (ERIC Document 
Reproduction Service No. ED 059 151) 

Manis, D., & Solomon, W. Using task analysis to generate teacher training 

objectives and activities and evaluate their adequacy . Paper presented at the 
meeting of the American Educational Research Association, Chicago, April 1974. 

132 

B-88 



erJc 



AUTHORING AID 
BIBLIOGRAPHY 



VOLUME I 



MDC E1570 
1 1 FEBRUARY 1977 



Course Management-Support Personnel, Facilities, and Equipment 

Defense Documentation Center. Government acronyms and alphabetic organizational 
designations used in DDC . Alexandria, Va.: Author, April 1975. National 
Technical Information Service Reproduction Service No. AD/A 006 800) 

Defense Research Analysis Establishment. Computer-aided sequential testing for 
equipment reliability . Ottawa, Ontario: Author, June 1973. (NTIS 
Reproduction Service No. AD 919 179) 

Resta, P. E., & Hanson, R. A. Installation requirements for the SWRL first year 
communication skills program. Evaluation data. 1968-69. Inglewood, Calif.:' 
Southwest Regional Educational Laboratory, June 1971. (ERIC Document 
Reproduction Service No. ED 057 993) 

Sullivan, D. J., & Smith, E. A. Instructional media carrel systems (Report No. 
AFHRL-TR- 72-48). Lowry Air Force Base, Colorado: Air Force Human Resources ■ 
Laboratory, Technical Training Division, February 1974. 

Course Management - Text-handling and Courseware Reproduction Systems 

Human Resources Research Organization. Project IMPACT: Computer-administered 
instruction: Preparing and managing the content of instruction . IMPACT 
text- handling subsystem. Alexandria, Va.: Author, September 1971. (ERIC 
Document Reproduction Service No. ED 055 450) 

National Security Industrial Association. The military technical manual industry . 
Washington, D.C.: Author, Logistics Management Advisory Committee, Technical 
Publications Panel, June 1972. 

Course Management - Organizational Interaction 

Andrews, F. M., & Pelz, D. C. Dimensions of organizational atmosphere (Analysis 
Memo No. 7). Ann Arbor: University of Michigan, Institute for Social 
Research, Survey Research Center, February 1961 . 
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Bidwell, C. E. The social psychology of teaching. In R.M.W. Travers (Ed.), 
Second Handbook of Research on Teaching . Chicago: Rand McNally, 1973. 

Dreeben, R. The school as a workplace. In R.M.W. Travers (Ed.), Second Handbook 
of Research on Teaching . Chicago: Rand McNally, 1973. 

Lortie, D. C. Observations on teaching as work. In R.M.W. Travers (Ed.), Second 
Handbook of Research on Teaching . Chicago: Rand McNally, 1973. 
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REFERENCE NUMBER: 3.3.1 REVIEW/SELECT EXISTING MATERIALS 

CITATION: 

Computer assisted planning: A user manual for computer-based resource units . 
Buffalo, New York: State University College Buffalo, Facultzy of Professional 
Studies, Research and Development Complex, October 1972. 

ABSTRACT: " 

This user manual describes computer-based resource units and gives directions 
for obtaining the units. Computer-based resource units are instructional packets 
consisting of objectives, content statements, activities, materials, and evaluative 
devices which are coded to various learner variables. A request form and a listing 
of approximately 40 unit titles are given. 

Listed with applicable grade levels, typical titles include: (a) Alcohol; 
(b) Career Education; (c) Communications; (d) Drugs; (e) First Aid and Survival ; 
(f) Illinois Test of Psycholinguistic Abilities; (g) Measurement; (h) Nutrition; 
(i) Safety Education; (j) Speaking and Listening; and (k) Mental Health. 

</■ . 

For group requests, up to five units may be requested. For individual requests, 
student-teacher planning is encouraged in order to select appropriate units. 
Student variables such as sex, reading level, mental age, chronological age, 
general interests, developmental tasks, and physical handicaps may be indicated on 
the request forms. When the resource guides have been received, an evaluation form 
included with the guide should be completed. 



Additional information may be obtained from: 

Research and Development Complex 
State University College at Buffalo 
1300 Elmwood Avenue 
Buffalo, New York 14222 
(716) 862-5433 
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Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



VOLUME I 

Computer-based 
Individualized Instruction 
None 

Civilian (grades K-12) 

3.3 Review/Select Existing Materials 

Tool 

9 

None 
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REFERENCE NUMBER: 3.3.2 REVIEW/SELECT EXISTING MATERIALS 

CITATION: 

. Department of defense audiovisual information system: A user guide for the audio- 
visuar products data base . Arlington, Va. : Department of Defense Directorate 
for Audio-Visual Activities, Office of Information for the Armed Forces 
(OASD/M&RA), October 1975. 

ABSTRACT: 

This detailed user guide is intended to provide Jhe required information for^ 
participation in the Defense Audiovisual Information Network, and to utilize the ' 
Audio-Visual Products Data Base in that Network. Instructions, procedures, and 
background information are provided that cover the initial delivery, installation, 
and start-up operations of terminal equipment used to remote- access the centralized 
audio-visual data files. The user's guide is divided into two sections. - 



The first section of the report gives general information including: 
(a) terminal installation of Hazeltine equipment ; (b) terminal operations of ^^^^^^ 
Hazel ti ne equi pment ; ( c) communi cati ons features , i nc 1 udi ng Wi de Area Tel ecommuni- 
cations Service (WATS) service numbers ; (d) a list of individuals and their : 
official respohsibilities for system management} and (e) a glossary of terms. 

The second section presents specific information about the: (a) concept of the 
system, including hardware and software configuration ; (b) file access procedures ; 
(c) retrieval processing; and (d) input processing, including operations of the 
on-line text editor and the off-line data preparation requirements. 

Several detailed appendices provide: (1) input data guidelines for DD Form 
1995, an official Department of Defense form for data coding; (2) data descriptions; 
(3) segment descriptions; (4) search instructions; (5) output options ; (6) on- and 
off-line tabulations; (7) tutorial messages for users; and (8) default conditions. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type:.^ 
Number of pages: 
Number of references: 



^Computer-based 

Conventional and individualized instruction 
None 

Military (Joint Service) 

3.3 Review/Select Existing Materials 

Tool 

150+ 

None 
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REVIEW/SELECT EXISTING MATERIALS 



Index of Army motion pictures and related audio-visual aids (Pamphlet No. 108-1). 
Washington, D.C.: Department of the Army, Headquarters, September 1974. 

Index of Army motion pictures for public non-profit use (Pamphlet No. 108-4). 
Washington, D.C.: Department of the Army, Headquarters, May 1975. 

ABSTRACT: 

The first report is a catalog wi,th annotations of official Department; of the • 
Army motion picture films and related audio-visual materials, including slides, 
audio-visual kits, recordings, and tape recordings. It is intended for use by 
instructors, information officers, and other film users (including defense contrac- 
tors) of the Army and active Reserves. Both classified and nonclassified 
materials are listed. 



The second report lists annotations for motion picture films for nonprofit use i 
by Government agencies, civic, religious, fraternal and educational groups, schools, 
colleges, universities, and other agencies. Use of portions of the films or of 
soundtracks only is prohibited, and no admission fee may be charged in connection 
with showings. Films are shipped free of charge, via parcel post, and the reverse 
side of the mailing card is franked in order that they may be returned free of 
charge. Films coded with an "S" in the annotated entries may be purchased by 
civilian organizations. 

The films and related audio-visual aids in the first volume are primarily 
oriented to Army career fields and specific activities. The films in the second 
volume are concerned with items of more general interest such as Civil Defense 
preparedness, information and education, sports, professional, medical, research 
and development films. State-by-state listings of Army Audio-Visual Support 
Centers are given in both volumes. 
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Audio-visual, transparencies, sound recordings, films 

Conventional and individualized instruction 

None 

Military (Anny) 

3.3 Review/Select Existing Materials 
Tool 

236 (first report), 50 (second report) 
None (both reports) 
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REFERENCE NUMBER: 3.3.4 REVIEW/SELECT EXISTING MATERIALS 

CITATION: 

National network for curriculum coordination in vocational-technical education . 
Washington, D.C.: Department of Health, Education, and Welfare, Office of 
Education, 1975. 

ABSTRACT: ^ . 

Development, publication, and distribution of curriculum guides and various 
curriculum support services are available from a national network of six regional 
curriculum management centers. 

Since all regional centers operate under the same charter from the Office of 
Education and provide essentially the same kinds of services, the Northwestern 
Vocational Curriculum Management Center, located at Olympia, Washington, may be, 
discussed as a typical center. One person appointed by the State Director of 
Vocational Education €ducBti on from each state served by the center provides a 
10-state Advisory Comnlittee. The Advisory Committee sets operating policy and 
emphasizes the inventorying and cataloging of curriculum and other vocational 
materials currently in use within the ten-state area. Information is also- 
collected about curriculum development activities currently underway so that 
curriculum developers may become aware of the work of others. A curriculum guide 
of materials available for distribution within the service area is periodically 
published. Curriculum entries in the guides provide information about the content 
of the curriculum material packages. Additional entries specify the use level 
(for example, "sec, postsec"), numbier and length of pieces in the package, when 
the package was developed, whether the material is listed in ERIC, whether it was 
field-tested, and ordering information, including cost. Curriculum materials are 
available in specific content areas as well as in more general areas such as 
individualized education, innovative practices, guidance, and teacher education. 
Lending library listings are also given. 

Curriculum guides are published in the form of brochures, booklets, and micro- 
fiche. The Northwestern Center curriculum guide is published In newspaper format. 
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Since curriculum material holdings of individual centers are too extensive to 
be usefully annotated here, curriculum guides and additional information may be 
obtained from: 



William Reynolds, Director 
Illinois Curriculum Mgt. Center 
Board of Voc. Ed. and Rehabilitation 
1035 Outer Park Drive 
Springfield, IL 62706 

Dr. Joseph Kelly, Director 
Voc, Curriculum & Media Research 
Division of Vocational Education 
225 West State Street 
Trenton, NJ 08625 

Dr. James Becket, Director 
Western Curriculum Coord Center 
Vocational Education Section 
Dept. of Education 
721 Capital Mall 
Sacramento, CA 95814 



Dr. James E. Wall, Asst. Dean (R&D) 
Mississippi State University 
P.O. Draw DX 

Mississippi State, MS 39762 

Mr. Ron Meek, Coordinator 
Curriculum Instructional Mtrls. Center 
State Dept. of Voc. and Tech. Ed. 
1515 West Sixth Avenue 
Stillwater, OK 74074 

Mr. James L. Blue, Director 

Voc. Curriculum Management Center 

Coordinating Council for Occupational 

Education 
Building 17, Airdustrial Park 
Olympia, WA 98504 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Print, microfiche 

Conventional and individualized instruction 
Individual curriculum packets may be field-tested 
Civilian 

3. 3 Revi ew/S'el ect .Exi sting Mat6ri al s 
Tool 

Varies with each center's publications 
Varies with each center's publications 
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REVIEW/SELECT EXISTING MATERIALS 



Special reserve components educational video tape catalog (Pamphlet No. 350-4). 
Fort Monroe, Va.: Department of ; the Army, Headquarters, United States Army . 
Training. and Doctrine Command, 25 March 1975. 

i 

ABSTRACT: 

This well-indexed pamphlet contains a listing of educational video tape 
programs specifically selected for unit training. Primarily intended for use by 
Reserve Components, it may be used by other services. Amy schools, and training 
centers. Video tapes requested by users will be copies furnished in 3/4-inch 
U-Matic video cassettes or in 1/2-inch reels for use with Sony Television Trainers.^ 
TRADOC/orm 517-R (contained in Appendix B) will be completed by the user and 
forwarded to the appropriate Training Aids Support Office (TASO). The addresses; V 
and support areas of specific TASOs are listed in Appendix A. Loans are made for ^ 
up to 60 days. A permanent retention; program (loans in excess of 60 days) allows 
copies of tapes to be made on user-subp lied raw cassettes/reels. Cost per 30- 
minute blank 3/4-inch cassette is approximately $25.10 and for a 30-minute 1/2-inch 
blank reel is approximately $23.50. I 

Listings of video tape programs are arranged by: (a) subject matter; 

(b) alphabetical indexp cross-referenced by title or subject-matter key word ; and 

(c) key-word in title, including more |than one key-word per title. Video tape 
program entries include the number, tiltle, subject and target audience (normally ^ 
by career specialty). | 

The subject matter of the video tape programs are: (1) Air Defense ; (2) Armor ; 
(3) Field Artillery; (4) Aviation; (5)| Engineer ; (6) Intelligence ; (7) Military 
Assistance; (8) Military Police; (9) Ordnance; (10) Quartermaster;. (11) Signal^ 
(12) Transportation; (13) Women's Arrny] Corps ; and (14) Non-Proponent. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Video tape 

Conventional and individualized instruction 
Individual programs may be evaluated 
Military (Army) 

3.3 Review/Select Existing Materials 
Tool 

200+ ^ 
None 
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REFERENCE NUMBER: 3.3.6 ' ocurr,, 

REVIEW/SELECT EXISTING MATERIALS 

CITATION: 

Vocational education. State instructional materials for office occupations . 
Washington, D.C.: U.S. Government Printing Office, 1973. 

ABSTRACT: 

An annotated listing of curriculum materials available from state education 
agencies is provided in order to assist planners, administrators, vocational 
educators, and course developers in obtaining existing curriculum materials. This 
particular title applies to office education. Other guides in the series are 
available in the following areas: (a) agriculture; (b) distributive education; 
(c) health occupations; (d) home economics; (e) technical education; and (f) trade 
and industrial occupations. They may be obtained from the Superintendent of 
Documents/ Government Printing Office, Washington, D.C. 20402. Requests fdr 
specific material in a guide should be addressed to the appropriate state education 
agency. 

The material in the guide is arranged by coded entry. The first 2-digit 
position! indicates the subject matter area (14, Office Occupations). The second 
2-digit position indicates the principal segment of subject matter (14.02, Business 
Data Processing Systems Occupations). The third 2-digit position indicates divi- 
sion of principal segment (14.0202, Peripheral Equipment Operators). The last 
2-digit position indicates the first-level detail of division of principal segment 
(14.020201, Key Punch and Coding Equipment Operators) . A discussion of the coding 
system is contained in: ' ' 

U.S. Department of Health, Education, and Welfare, Office of 
Education. Vocational Education and Occupations (OE-8 0061). 
Washington, D.C: U.S. Government Printing Office, July 1969. 

Curriculum materials listed in this particular guide include: (a) Accounting 
and Computer Occupations (Bookkeeping); (b) Business Data Processing Systems 
Occupations (Key Punch and Coding Equipment Operators, and Other); (c) Filing, 

Office Machines, and General Office (File Clerks, General Office Clerks, and 
Other) ; (d) Steriographic, Secretanal , and Rela^^t^ 

* ' B-101 

■ 145 ,^ V 



AUTHORING AID 
BIBLIOGRAPHY 



VOLUME I 



MDC El 570 
M FEBRUARY 1977 



Others) ; (e) Supervisory and Administrative Management Occupations (Budget 
Management Analysis); (f) Typing and Related Occupations (Typists and Other) ; and 
(g) Office Occupations, Other. 

There are two additional sections in the guide. The first contains a listing of 
more general manuals and courses useful to administrators and teachers, such as 
New Jersey's Student's Manual for Supervised Teaching of Vocational Subjects and 
Oklahoma's Procedure f or Establishing Area Vocational Schools . The second contains 
a state-by-state listing for ordering procedures. Some state education agencies 
charge for publications ; others sehd,,one complimentary copy of a curriculum pack- 
age, and others send everything free. . Curriculum developers in a particular state 
should check that state's order requirements. 

DATA SUMMARY: 

Media: Printed, programmed text, arid audio-visual 

Application: Conventional and individualized instruction 

Evaluation: Individual states may evaluate their materials 

Environment: Civilian 

Key number: 3.3 Review/Select Existing Materials 

Authoring aid type: Tool 

Number of pages: 25 

Number of references: None 
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REFERENCE NUMBER: 3.4.1 



DEVELOP INSTRUCTION 



CITATION: 



Best, E. The develo pment of visual materials for CAI (Technical Memo No. 5). 
Austin: The University of Texas at Austin, Computer-Assisted Instruction 
Laboratory, August 1971. 



Various procedures and materials used to minimize expense and time in the 
preparation and production of computer-controlled films are discussed. The films 
were intended to be used on a computer-controlled rear screen projector which 
, could contain up to 1,022 different frames in either color or black and white. 
Frames were addressable in the standard sound track area. For text-typed, one- 
color, 20-frame filmstrips, the cost of production was approximately $97 and 
processing time was two to three weeks. 

The first step in the production of the filmstrips was the decision by the 
author of what portion of his program to display on th.<>„„terminai output device and 
what portion to display from the rear screen projector. The next steps were to 
meet with a media specialist, discuss possible layouts, and approve a text-frame 
master for the preparation of 8 x 10-inch black-and-white negatives. Author 
approval for the negatives was required, for filmstrips, once produced, are .diffi- 
cult to change. 

Text masters were prepared on frosted acetate. Artwork masters were done on 
8 x 10-inch poster board, and photographed without enlargement. Typing was done 
in an area 4-1/2 x 6-1/2 inch, which enlarged proportionately to 8" x 10" with a 
1/2-inch to 3/4- inch border. Typing compared favorably with pressure-sensitive 
and photocomposed letters in terms of cost and speed of production. It was noted 
that no more than nine lines of type should be put on a master if fast reading 
is expected. 

Once the 8" x 10" c*o1or-backed negatives were obtained, they were photographed 
in sequence on a light-box. Inexpensive and easily built, the light-box and its 
materials and dimensions-are~discussed-in -the text. The light-box was also used 



ABSTRACT: 
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for alignment of the negative on i clear piece of acetate. A registration pen bar 
was taped to the top of the box, and an Acme field chart was used to align the 
negative on the acetate. Tens assured squared and centered frames, which were 
essential for accurate photography. 

For local production shops, this report provides much information required for 
processing filmstrips into slides for either computer-controlled rear screen 
projectors or for transparencies. 



DATA SUMMARY: 
Media: 

Application: 

Evaluation: 

Environment: 

Key number: 

A/' v^crjirii; aid type: 

Number of pages: 

Number of references: 



Filmstrips, slides 

Conventional and individualized instruction 

None for procedure but materials used were evaluated 

Civil ian 

3.4 Develop Instruction 

Tool 

12 

None 
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REFERENCE NUMBER: 3.4.2 DEVELOP INSTRUCTION 

CITATIONS: 

Deterline, W. A., & Lenn, P. D. Coordinated instructional system. Lesson book , 
Palo Alto, Calif.: Sound Education Inc., 1972. 

Deterline, W. A., & Lenn, P. D. Coordinated instructional system. Study resource 
materials book . Palo Alto, Calif.: Sound Education Inc., 1972. 

ABSTRACT: 

This self-study course teaches a set of skills and techniques called CISTRAIN 
for designing, developing, and implementing instruction. CISTRAIN means Coordi- 
nated instruction ^stem for Training. During the full course of instruction, 
the istudent prepares two lessons tn his subject matter area and tests them on 
other students. Course completion times range from 20 hours to 50 hours. Average 
completion time is^35 hours but one significant factor affecting that time is the 
size and scope of the two lessons prepared by the student. 

The Study Resource Materials Book contains all instructional articles, sample 
lesson materials, and procedural guides necessary for completing the course. The 
Lesson Book guides the student through the appropriate portions of Study Resource 
Materials Book and provides reinforcement when required. Both books are concerned 
with three processes of instructional design and development: (a) lesson develop- 
ment including testing instruments ; (b) evaluation and tryout of the lesson j and 
(c) selection of the correct medja for lesson presentation. 

Lessons are also called study units. Evaluation of student performance on 
each study unit depends upon a humar monitor (called an Enabler) who conducts 
group discussions and peer critiques. There are six components to each study unit. 
First is an introduction that discusses the topics, scope, and purpose of the 
lesson. The next component contains descriptive objectives that discuss what the 
student should be able to do at the end of the study unit. Study resources and 
directions are then provided that indicate what the student is to do, what 
resources are required, and when and how the instruction is to be carried out. 
Next are practice exercises which help students apply what they have learned in : 
the lesson. Questions are asked^^^^^^^^^^^ 
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of the practice exercise results is the next component of a study unit*. Correct 
responses for both questions and tasks are shown, and explanatory feedback is 
given. The last component of a study unit is actual self-evaluation and 
discussion, which basically provides further practice and more detailed discussions. 

A glossary of 29 terms is included. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 



Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Programmed text 

Individualized instruction 

"The course materials underwent three tests and 

revisions, and more than one hundred faculty and 

administrators from California Community Colleges 

participated in the developmental tests The final 

revision was based on data from the final field tryout." 
Civilian 

3.4 Develop Instruction 
Tool 

106 ( Lesson Book ), 94 ( Study Resource Materials Book) 
None (both books) 
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REFERENCE NUMBER: 3.4.3 DEVELOP INSTRUCTION 

CITATION: 

Guidelines for developing computer based resources units. Revised , Buffalo: 
State University of New York, Educational Research and Development Complex, 
March 1972. (ERIC Document Reproduction Service No. ED 073 586) 

ABSTRACT: 

Guidelines for writing computer-based resource units are presented and 
organized into two operations. The first operation involves formulating behavioral 
objectives, instructional content, activities, materials, and measuring devices 
for each unit of instruction, which is termed a "computer-based resource unit." 
The second operation involves coding these five component items for future computer 
retrieval^ Every content item, activity, material, and measuring device is coded 
to a behavioral objective, and each behavioral objective is coded to several 
learner characteristics. 

Specifications are given for writing objectives, coding components to objec- 
tives, and coding objectives to learner variables. Objectives written for 
computer-based resource units must: (a) be behavioral with measurable perfor- 
mance; (b) be oriented to the grade level of the student; (c) be relevant, based 

^pn J'spme soci^^ demand, student need, or phi losophi cal imperati ve; (d) cal 1 f or- 

a single behavior in order to minimize confusion in the evaluation procedure; and 
(e) yet be general enough so that different approaches may be made for accommo- 
dating individual differences in students. Coded components have unique identifi- 
cation numbers, beginning with *'V' and continuing in consecutive fashion. For 
coding objectives (and thus the instructional material coded to each objective) to 
learner variables, an appendix of fourteen variable categories containing 248 
instructional variables is given. The fourteen variable categories are: 
(a) student interests; (b) sex; (c) developmental tasks; (d) major social func- 
tions, i.e., "those societal conditions in which all individuals must participate 
at some time in their lives;" (e) reading level; (f) instructional activity; 
(g) suggested approach to instruction; (h)taxonomical sophistication of the 
objective; (i) material descriptor; (j) instructional grouping, i .e. , "the one 
designation which would provide the most effective environment for learning;" 
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(k) evaluation devices; (1) mental age of the student and chronological age of the 
material (its "social appropriateness"), which may differ according to the specific 
material ; (m) physical handicaps; and (n) learning environment, such as classroom 
or field trip. 

In addition to the appendix mentioned above, three other appendices provide: 
(a) a coding sheet for objectives; (b) appropriate categories for coding component 
parts of a computer-based resource unit; arid (c) a coding sheet for learner 
variables. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Computer-based 
Individualized instruction 
None 

Civilian 

3,4 Etevelop Instruction 

Tool: 

33 

None 
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REFERENCE NUMBER: 3.4.4 



DEVELOP INSTRUCTION 



CITATION: 



How to design and produce individualized instruction programs, St. Paul, 

Minnesota: Minnesota Mining and Manufacturing Co., Visual Products Division, 



This pragmatic workbook is intended to be used for developing audio-visual 
programs for the 3M Sourid-On-Slide System. With minimal instructional theory, the 
following sample worksheets and forms are given: 



1. Task Listing . Task Listing Sample, Action Verbs Sample, Task 
Listing Worksheet; 

2. Task Analysis , Task Analysis Sample, Task Analysis Worksheet; 

3. Performance Objectives . Performance Objectives Sample and Worksheet; 

4. Evaluation Checklist , Evaluation Checklist Sample, Evaluation 
Checklist Worksheet; 

5. Learning Activities Guide . Learning Activities Guide Sample, 
Learning Activities Guide Worksheet; 

6. Sound-On-Slide Unit Design and Script . Story Board (Picture 
Descriptions), Story Board Worksheet, Completed Story Board; 

7- Self-Evaluation . Instructional Systems Development Evaluation Form, 
Individualized Instruction Materials Development Evaluation Fonii, 
Package Objectives Evaluation Form; and, 

8. Development Schedule. Instructional Materials Development Schedule 
Worksheet. 



In keeping with the no-frills approach, the job area used as an example for 
the development steps listed above is Service Station Attendant. 

DATA SUMMARY: 



1972. 



ABSTRACT: 



Media: 



Audio-visual 



Appl ication: 

Evaluation: 

Environment: 



Individualized instruction 

None 

Civilian 
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Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



3.4 Develop Instruction 

Tool 

23 

None 
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REFERENCE NUMBER: 3.4.5 DEVELOP INSTRUCTION 

CITATION: 

Nesbit, M., & O'Neil, H. F., Jr. Guidelines for editing programmed instruction . 
Austin: University of Texas Press, 1976. 

ABSTRACT: . 

Course authors need two kinds of information. First, to communicate with 
instructional designers and educational psychologists, course authors need 
accepted definitions for types of frames and rules used in programmed instruction 
(PI).. Second, to write effective programs, course authors need specific guide- • 
lines for separating lesson content into rules and examples which can be integrated 
into a series of logical frames. Nesbit and O'Neil present both kinds of 
information. ' 

The report's first two sections define 23 basic terms of PI (such as frame, 
step size, and fading) and discuss threis models of PI (linear, branching, and 
autoelucidative) . In linear programs, all learners proceed sequentially through 
the same content in the same way; hence, individualization occurs according to the 
rate of student progress through the program. In branching programs, learners 
proceed along individual paths determined by answers to multiple-choice questions. 
Right answers move a student forward; wrong answers move him backward or forward to 
reinforcement frames or remedial material. In autoelucidative programs, all learners 
progress through linear steps that also contain branching opportunities. Large step 
sizes, i.e.v large amounts of text in single frames, and multiple-choice questions, 
allow students 'I'd" progress quickly for correctly answering questions or to reread 
summaries of specific material for incorrectly answering questions. The authors 
present diagrams and examples for each type of model. 

The third section. of the report presents guidelines for editing PI text which 
allow authors. to check lesson content and style for logical consistency and to 
tailor programs to the student population. For example, it is suggested that: 
(a) typed lines should not be more than ten words or 4-1/2 inches long; (b) defi- 
nitions should be separated for emphasis; and (c) testing should be confined to 
the major issue of the frame. 
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Examples of good and bad frames are also given. Two typical errors in frame 
design are illustrated: (a) a student is not required to respond to all material 
in a frame, i.e., too narrow a response; and (b) a student is given too much 
material to "digest*' at once, i.e., too broad a stimulus. The authors also 
caution against over- or under-prompting, using page numbers as multiple-choice 
question alternatives that inadvertently cue the right answer, and using indis- 
criminate fading, which leads to memory, not understanding. 

DATA SUMMARY: 

Media: Programmed text 

Appl ication: Individualized instruction 

Evaluation: None 

Environment: Military (Air Force) 

Key number: 3.4 Develop Instruction 

Authoring aid type: Tool 

Number of pages: 35 

Number of references: 14 annotated references 
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REFERENCE NUMBER: 3.4.6 1 ' INSTRUCTION 

CITATIONS: 

Siegel, A. I., Federman, P. J., & Burkett, J. R. Increasing and evaluating the 
readabilit y of Air Force written matierials (Report No. AFHRL-TR-74-28) . 
Lowry Air Force Base, Colorado:. Airj Force Human Resources Laboratory, 
Technical Training Division, August |1974. 

Siegel, A. I., Lambert, J. V. , & Burkettj, J. R. Techniques for making written 
material more readable/comprehensible (Report No. AFHRL-TR-74-47) . 
Lowry Air Force Base, Colorado: Airj Force Human Resources Laboratory, 
Technical Training Division, August i]974. 

ABSTRACT: | 

In a previous survey of readability iresearch, Siegel, Federman, and Burkett 
(1974) synthesized the findings most useful to writers. In the second report, 
Siegel, Lambert, Burkett (1974) illustrated the findings with very good examples. 
The topics covered include: (a) morphemes (the number of sounds within words) ; 
(b) center-embedding (putting a clause between the subject and predicate of a 
sentence); (c) sentence depth; (d) vocabulary diversity; (e) linking stated 
thoughts with intended conclusions; (f) seeing generalizations; (g) cutting out 
unnecessary facts in diagrams and charts; (h) repeating facts; (i) organizing 
-material ; and (j) avoiding abbreviations.; Several examples will indicate the 
usefulness of the report. 

■ 

Morphemes. The more morphemes contained within a word, the iharder the word is 
to understand. Instead of "disassemble the linkage," write "take apart the 
linkage." 

Center-embedding. Instead of "The president, having no intention of yielding " 
to the repeated demands of the citizens, was forcing them to revolt," write "The 
president was forcing the citizens to revolt because he had no intention of yield- 
ing to their repeated demands." 
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Sentence depth . Breaking a long sentence into shorter sentences increases 
readability. Instead of "The new car which the neighbors bought and which has 
already been delivered, is a gift for their son who is graduating from college 
this semester," write "The new car has already been delivered. It was bought by 
the neighbors as a gift for their son. He is graduating from college this 
semester." 

Vocabulary diversity . Find the words you want to use and stick with them. 
A thesaurus is heavy in more ways than one. "One little piggy went to market, 
one little piggy stayed home" is less burdening then "One little piggy went to 
market, one diminutive porkster stayed home." 

Linking . Don't force your reader to make his own conclusions or to make a 
guess as to what goes with what. Instead of "Deliver the beer and the milk to the 
NCO club and the commissary," write "Deliver the beer to the NCO club and the milk 
to the commissary. " 

Seeing generalizations . In theory-oriented instruction, make generalizations 
when necessary. Instead of simply saying "The voltage in a circuit is equal to 
the product of the intensity of the current and the resistance of the current," 
add "Therefore, the voltage can be increased by decreasing resistance or by 
increasing current intensity." 

Repeating facts . Important facts should be repeated. Instead of saying 
"After you have tightened the three wing-bolts, attach the antenna" and then 
dropping it, write "After you have tightened the three wing-bolts, attach the 
antenna. Tightening the three wing-bolts before antenna-attachment simplifies 
later steps. If you fail to tighten the three wing-bolts before antenna- 
attachment, the result will be..." 

Organizing material . Provide memory aids for your reader whenever possible. 
Acronyms are memory aids (RADAR for Radio Detecting and Ranging). Another type 
of memory aid is Psalm 145, where the first letter of each line forms the alphabet 
sequence. A third type of memory aid is rhyming or jingles such as "Hey diddle 
diddle, the cat and the..." 
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Avoiding abbreviations . Excessive use of abbreviations can paralyze our 
ability to think. Instead of "The STS, POI, and OSR are useful for TA," write 
"The Specialty Training Standard, Plan of Instruction, and Occupational Survey 
Report are useful for task analysis." 



DATA SUMMARY. 
Medi a: 

Appl ication: 
Evaluation: 
Envi ronment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references: 



Print 

Conventional and individualized instruction 
None 

Military (Air Force) 

3.4 Develop Instruction 

Tool (second report) 

88 (first report), 28 (second report) 

35 (first report), 1 (second report) 
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REFERENCE NUMBER: 3.4.7 



DEVELOP INSTRUCTION 



CITATION: 

Thiagarajan, S. The programming process: A practical guide . Morthington, Ohio: 
Charles A. Jones Publishing Co., 1971. 



Thiagarajan divides the programming process into five stages: task analysis j 
design; editing; developmental testing; and validation testing. In addition to 
those stages, this self-paced guide to programming printed texts also considers 
effective behavioral objectives and various kinds of testing instruments. Com- 
pletion time for the program is estimated to be 40 hours. The last frame of each 
chapter requires the student programmer to apply the concepts learned in the 
chapter and analyze a topic of his choice. He is required to design, develop, and 
validate a program on that topic. The resulting program is evaluated by using 
specific checklists found in either the main part of the book or in the confirmation 
section. The total number of points that can be awarded' for the assignments is 100, 
and students scoring 95% or more receive an "A" while those scoring between 90% and 
94% receive a "B". No grades are indicated for other scoring ranges. 

A Magic Square is used as a consistent example -for explaining task analysis, 
behavioral objectives, and criterion tests (including entry tests, pretests, and 
posttests). A Magic Square is a multicellular square containing an odd number of 
cells with numbers placed in them such that each horizontal row, each vertical 
column, and each diagonal sum to the same number. The numbers may be either 
consecutive or increased by increments of five. (If you want to know how to con- 
struct a Magic Square, check p. 18 of the text!) Since a set of rules and pro- 
cedures exist for creating the product, a task analysis of a Magic Square.-can be 
done. Thiagarajan explains task analysis. Since there is an intended visible out- 
come, behavioral objectives for a Magic Square can be constructed. Thiagarajan 
explains behavioral objectives. Finally, since there are correct Magic Squares 
and incorrect Magic Squares, criterion tests can be written for Magic Square con- 
struction. Thiagarajan explains that also. It is a novel way to explain task 
analysis, behavioral objectives, and criterion tests , but the intricacies of the 
Magic Square sometime create confusion. 



ABSTRACT: 
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Editing frames, developmental testing and revision, and validation testing 
are discussed without using the Magic Square. The discussion on editi,ng is directed 
toward external language and subject-matter editors to which the student-programmer 
presumably has access. Emphasis is given to three types of editing: (a) revision 
of subject matter involving changes in language and frame sequence} (b) revision 
of language not involving changes in subject-matter or frame sequence; and (c) 
revision of program involving changes in language but not in subject matter. In 
the developmental testing and revisions, importance is placed on introductory 
comments which help reduce students' test anxiety, aversion to criticism, and 
aversion to self-instruction. Validation testing, it is noted, collects and 
evaluates four kinds of data: (a) description of the students involved in 
validation testing; (b) indication of gain or loss in the students' achievement; 
(c) time taken by the students to complete the program; and (d) the instructional 
situation in which the validation testing took place. 

A confirmation section at the conclusion of the book contains correct responses 
to the questions embedded in the text. 

DATA SUMMARY: 

Media: Programmed text 

Application: Individualized instruction 

Evaluation: 32 students at Indiana University 

Key Number: 3.4 Develop Instruction 

Authoring aid type: Tool 

Number of pages: 122 

Number of references: 11 
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REFERENCE NUMBER: 3.5.1 VALIDATE INSTRUCTION 

CITATION: 

Abedor, A.J. Development and validation of a model for formative evaluation of 
self-instructional multi-media learning systems . Paper presented at the 
meeting of the American Educational Research Association, Chicago, April 1972. . 

ABSTRACT: 

A revision model was developed and field-tested in order to: (a) identify 
instructional deficiencies through data collection; (b) analyze the deficiencies 
which require revision of materials; and (c) design, integrate, and evaluate the 
revisions. The study was conducted in four phases. The first phase was design 
of a MK I flowchart revision model based^on a review of the literature. The 
second phase was tentative evaluation of the M( I model based on interview's with 
several developers of multi-media instructional treatments. The third phase was 
to incorporate the results of the interviews into a KM II revision model which 
used small -group tryout and debr\,efing procedures as the primary technology for 
identifying instructional problems and recommending effective revisions. The 
fourth phase was empirical testing of the MK II revision model in three separate 
experimental situations. 

Abedor notes that revision of materials used in a multi -media instructional 
system can be a monumental task, and that persuasive data (in terms of both quantity 
and quality) are needed to convince developers that revision is required. "In 
other words, the delivery vehicle must have several serious discrepancies before 
it warrants an overhaul." He also notes that developers usually do not have the 
time or resources to make multiple, iterative revisions of materials. Therefore, 
the MK II model was designed to generate large amounts of corroborative data on 
instructional problems in a one-shot trial of the prototype lesson. 

Unlike other formative evaluation models, this model uses face-to-face 
interviews between the lesson developer and small groups of students. The optimal 
size of the group was determined to be nine students plus the lesson developer. 
The procedures for group debriefings included the following: (a) selection and 
orientation of the volunteer students who vary in their entering abilities} (b) 
individual use of the prototype lesson materials by these volunteers; (c) 
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administration and assessment of learning and attitudinal measures to provide 
data for the debriefing (d) development of a debriefing document; and (e) imple- 
mentation of the group debriefing. 

Empirical testing of the MK II model was conducted in five field experiments 
with three Michigan State University faculty members and students in their courses. 
The validity of the MK II model was defined as enabling the lesson developer to 
identify unsatisfactory sequences of instruction, and as resulting in development 
of substitute, satisfactory sequences. If one-third or more of the group of 
students indicated dissatisfaction, boredom, confusion, or lack of understanding 
with a' particular sequence, then it was deemed. "unsatisfactory." Experimental and 
control groups were used to validate the model. 

In two experiments, significant differences were obtained (p. <01) favoring the 
lesson revised according to the model on all four dependent measures (posttest, 
gain score, percent achieving 80% criterion, and student attitudes). In the third 
experiment, a significant difference (p. <01) was noted only on the posttest. 

The author notes that, because of social and psychological interactions, the 
'overall effectiveness of the MK II for data gathering will perhaps vary from 
situation to situation. For example, "in the case of developer B, the prototype 
lesson was so ineffective and the derogatory comment of students so devastating 
that by the end of the debriefing he was simply unwilling to continue the process 
for the seemingly ^ungrateful students... In the case of developer C, he appeared 
unwilling or unable^^Thandle the relatively large number of derogatory conments. 
Thus, he closed off discussion prematurely and refused to revise the prototype 
lesson." In another instance, the author notes that the "groups appeared to 
develop a 'momentum' phenomenon. If they got started on a deragatory theme 
they kept going and the comments became increasingly derogatory until the author 
was forced to j)ecome defensive and begin justifying the lesson rather than 
exploring ways to improve it." Since the effectiveness of the MK II model depends 
on the data gathered during group debrief ings, the serious disruptions undermined 
the process. Whether these weaknesses are a fault of the methodology or of the 
group dynamics was not determined. 
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The u§'e of Abedor's model should be explored since, unlike most models, it 
attempts to function in a real-world, operational environment. Further, data 
seem to indicate that it works effectively in identifying and generating useful 
revisions. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references; 



None 

Conventional and individualized instruction 
None 

Civilian 

3.5 Validate Instruction 

Procedure 

45 

17 
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REFERENCE NUMBER: 3.5.2 



VALIDATE INSTRUCTION 



CITATION: 



Aleamoni, L.M. MERMAC: A model and system for test and questionnaire analysis 
(Research Rep. 330)'. Champaign: University of Illinois, Office of 
Instructional Resources, Measurement and Research Division, March 1970. 
(ERIC Document Reproduction Service No. ED 055 097) 



ABSTRACT: 



An evaluation model was developed and implemented which provides instructors 
with basic data required for decision-making about student abilities, course 
attitudes, grade distributions, and test appropriateness. The data are provided 
in useable formats and g|ve quick feedback to students. Specifically, the model 
is useful for test analysis and course evaluation questionnaires by allowing the 
user six options. ^ 

First, test items are scored. Up to forty subscores may be produced for each 
individual, and each item and response may be weighted either the same or 
differently for any subscore. A sample output from this SCORE program is given 
in the text. 

Second, scores for groups of individuals may be analyzed and the following 
descriptions produced: (a) frequency distribution and histogram; (b) mean, median, 
and standard deviation', (c) Kuder-Richardson reliability; (d) standard error of 
measurement; (e) Spearman-Brown prophesy for .90 reliability; (f) raw scores, 
standard scores may be weighted, and the sum assigned a letter grade. A sample 
output from this TOTAL program is given in the text. 

Third, a test item analysis printout may be given to the student. His test 
score, a list of items missed with his i^esponses, and the correct responses are 
printed. A sample of this ERROR program is given in the text. 

Fourth, item analysis data are provided as a plot of the percentage of 
individuals responding to the correct response by fifths of the total. Item 
alternative analysis data are provided as a proportion of individuals responding, 
a point biserial correlation, and the number responding to each alternative by 
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fifths of the total. A sample output of this ITEM program is given in the text. 

Fifth, item data may be analyzed according to some external criterion other 
than the keyed test score. The output format of this SELECT program is the same 
as the ITEM format. 



Sixth, data from course evaluation questionnaires, attitude scales, and other 
measures with no known correct response may be summarized. A frequency distribution 
of responses, a weighted mean, and a standard deviation for each item are provided. 
Subscores may be generated with means, standard deviations, split-half reliabilities, 
and percentage of individuals responding to the noncriterlon items. A sample out- 
put from this QUEST program is given in the text. The QUEST program has been 
applied to the Illinois Course Evaluation Questionnaire, which yielded responses to 
fifty items and provided six subscores concerning general course attitude, instruc- 
tional method, course content, interest-attention, instructor effectiveness, and 
unique items requested by a user. 

The MERMAC manual and system documentation can be obtained from the University 
Press, University of Illinois, Urbana, Illinois 61801. 



DATA SUMMARY: 

Media: Computer-based 

Application: Conventional and individualized instruction 

Evaluation: None 

Environment: Civilian 

Key number: 3.5 Validate Instruction 

Authoring aid type: Tool 

Number of pages: 14 



Number of references: 1 
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REFERENCE NUMBER: 3.5.3 VALIDATE INSTRUCTION 

CITATION: 

Bloom, B.S. Learning for mastery. Evaluation Comment . 1968,1(2) 

Hummel -Rossi , B. A formative evaluation of Bloom's mastery learning concept . 
New York: Van Valkenburgh, Nooger, and Neville, 1972. 

ABSTRACT: 

This two-part evaluation exercise is based on the article by Bloom, and is 
conducted in accordance with the twenty-item, multiple-choice test by Humnel-Rossi . 
The test is designed to assess understanding of the concepts given in Bloom's ar- 
ticle and understanding the use of formative evaluation. 

The article discusses the variables present in a mastery learning strategy, 
and the approach used by Bloom to derive a mastery learning strategy. There are 
five variables in a mastery learning strategy: (1) aptitude for particular kinds 
of learning; (2) quality of instruction; (3) ability to understand instruction; (4) 
perserverance ; and (5) time allowed for learning. Bloom outlines the preconditions 
necessary for mastery learning, the operating procedures required, and the outcomes 
to be evaluated. One important precondition is a perfonnance standard by which to 
judge the achievement of students. Another important precondition is formative 
evaluation, accompanied by diagnosis and followed by prescription. Outcomes may 
be measured in both cognitive and affective domains. 

The test is composed of an instruction sheet; a specially-treated answer 

sheet which, when rubbed, displays the answer; and the testing instrument. The 
answer sheet (or card) provides immediate feedback. If the wrong answer appears 
in the spot being rubbed, then alternative choices are rubbed until the correct 
answer appears. The whole testing package is called a "Trainer-rester Self-Scoring 
Student Response Device." A reprint of Bloom's article is included in the 
package. 

One's belief in the competency of course developers and evaluators is usually 
a matter of faith. This is one example of an instrument which may provide some 
data concerning that competency. 
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Print, chemically-treated answer card 
Conventional and individualized instruction 
None • • • 

Civilian 

3.5 Validate Instruction 
Tool 

11 (article), 4 (test) 
18 (article), none (test) 
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REFERENCE NUMBER :^ 3.5.4 VALIDATE INSTRUCTION 

CITATION: 

Rayner, G. T. An empirical study of a methodology for the revision of systemati- 
cally designed educational material (Report No. 24). Tallahassee: Florida 

State University, Computer-Assisted Instruction Center, March 1972. (ERIC 
Document Reproduction Service No. ED 067 877) 

ABSTRACT: 

A revision model for systematically designed educational materials was developed 
and tested. The model can be applied to a course which: (a) has been developed 
using some systems model; and (b) includes several subcomponents, including 
specific objectives clustered in some "unitized" way (conmonly called modules), 
instructional materials, criterion-referenced diagnostic tests, remedial prescrip- 
tions, pretests, and posttests. These components furnish the information needed 
for effective revisions. Data sources include both students and subject matter 
experts. Measuring instruments consist of content examinations, attitude scales, ! 
and collections of informal and solicited student comments. 

Application of the model results in content revisions and procedural revisions. 
Content revisions can be made on any or a]! of six components of instruction noted 
. above. A change in any one component may , necessitate changes in other components. 
P'^ocedural I'evisions include all operat pro- 
viding instruction to students. Attention is focused on problems identified by 
student criticisms. Procedural revisions are mainly made in pretesting, student 
orientation, instructional operation, posttesting, and grading. 

The revision model was applied to a required course in health education for. 
elementary education majors at Florida State University. Students were enrolled 
in the course for two successive quarters, and were randomly assigned to either 
of two treatments: computer managed instruction (CMI) ; or conventional instruction 
(CI). The CMI treatment was individual ized while the CI treatment was group-paced 
and managed by a professor. In both treatments, modularized instruction consisted 
of objectives, sample test items, selected instructional materials, diagnostic 
tests, remedial prescriptions, and posttests. During the first administration of 
the course, only 17 percent of the students reached criterion of 80 percent or 
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more. After the course was revised according to the modfil discussed here, 71 
percent of the students reached criterion. No significant attitude changes 
occurred across the administrations. Course completion times and total time 
signed-on to the computer terminal increased after the revision, even though 
the number of course objectives dropped from 32 to 23. The number of minutes 
logged-on to the terminal versus criterion performance was analyzed. Of those 
students spending 200 or more minutes on the terminal, 92% reached criterion; 
of those spending between 100 and 200 minutes on the terminal, 56?^ reached criter- 
ion; of those spending 50 to 100 minutes on the terminal, 73% reached criterion. 

After two successive administrations of the health education course, it 
was recormended that the principal content change in the model itself should be 
establishing more stringent criterion for test item revision. Since the overall goal 
was 80 percent mastery, test items with a posttest difficulty level below 80 
percent should be reviewed for ambiguity, isomorphism with the objective, clarity 
of related instructional material, accuracy of prescriptions, and comments from 
students who missed the item. Three procedural changes were also recommended: 
(Ij within-course reviews; (2) small-group help sessions; and (3) intra-course 
deadlines to prevent students from leaving the entire course to the last few 
days or weeks. The need for the last change was indicated by the fact that, in 
the second administration of the course, 34 percent of the tests were taken in 
the last week. 

Debriefing questionnaires for the CMI and CI groups are given in two appendices. 
DATA SUMMARY: 

Media: Computer -assisted instruction 

Application: Conventional and individualized instruction 

Evaluation: None 

Environment: Civilian 

Key number: 3.5 Validate Instruction 

Authoring aid type: Procedure 

Number of pages: 163 

Number of references: 29 
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REFERENCE NUMBER: 3.5.5 



VALIDATE INSTRUCTION 



CITATION: 



Ward, M. E., Examination and application of formative evaluation for author 
utilization during the preparation of a CAI course (Doctoral Dissertation, 
University of Pittsburg, 1972). (ERIC Document Reproduction Service No. 
ED 076 056) 



Ward investigated an author's use of formative evaluation during the prepara- 
tion of a CAI course. Four specific questions were examined: (1) what models 
exist for formative evaluation related to the development of course materials; 
(2) of those models, which one is more appropriate for use during CAI course 
preparation; (3) using the selected model , what results obtain for formative 
evalu^iori' during CAI course preparation; and (4) what guidelines and forms can 
be identified for inclusion in an author's manual for formative evaluation of the 
materials development process? 

Five evaluative models were studied. The first model, developld' by Stake, was 
based on: (a) descriptions of what intended antecedents or entry behaviors were 
expected, what intended instructional processes were planned, and what outcomes 
would be evaluated; (b) descriptions of what actually occurred versus what was in- 
tended to occur; and (c) two evaluative judgements, one subjective by an evaluator's 
personal standards and the other objective by comparison of the particular program' 
to alternative, similar programs. Briggs proposed a second model, the procedures 
of which would take effect during the development and evaluation phases of the 
material development cycle. Instructional materials would be selected and modified 
on the basis of: (a) learner characteristics} (b) intended outcomes ; (c) learning 
alternatives such as dual-track programs; and (d) performance test results from 
small-group tryouts and classroom tryouts. In a third model. Baker and Schultz 
outlined seven steps, or components, in the instructional development process. 
They are: (1) formation; (2) instructional specification; (3) item tryoutj (4) - 
product development; (5) product tryout; (6) product revision; and (7) operations 
analysis, in order that effective procedures may be clearly identified. The fourth 
model, by Abedor, (Reference 5. 5. 1 ) relied on technical review by subject matter ex- 
perts, group tryouts, and face-to-face debriefings between students who took the les 
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and authors who developed them. Glass proposed an evaluation model for finished 
products. The model , prepared -for use by outside evaluators in evaluating finished 
instructional products already on the market, considered nine points: (1) the pro- 
duct description; (2) goals evaluation; (3) point of entry of the evaluatorJ (4) 
trade-offs; (5) comparative cost analysis; (6) evaluation of technical quality, 
content, medium and availability; (7) evaluation of outcomes; (8) summative judge- 
ments and recommendations; and (9) scope and value claims modifying the summative 
judgements. 

After consultation with the dissertation committee. Ward decided to follow the 
Baker and Schultz model and add to it an outside evaluation component at the 
Instructional Specification stage. The model was applied during the development 
by the author of a CAI course for regular classroom teachers in education of 
visually-handicapped children. The course was delivered to rural school teachers 
via a mobile CAI laboratory. Early evaluation comments indicated the need for a 
handbook to accompany the on-line lesson materials, and so a handbook was written. 
Fifteen college students comprised the field test for the course. Revisions to the 
on-line course and the handbook were based on: (1) student response records; 
(2) on-line time records for each chapter and for the total course; (3) print-outs 
of unanticipated responses to course questions; (4) print-outs of student comments 
made on-line by students who had branched themselves into the comment routine; and 
(5) results of the final examination. The- Operations Analysis phase of the model 
identified points where critical time delays occurred, where problems developed, 
where specific activities facilitated the development process, and what forms and 
procedures would be useful for transfer to other developmental projects. 



"may 



Four procedures and forms were identified which, according to the author', 
be useful to other CAI course authors preparing course material for the [IBM] 
1500 system while at the same time conducting formative evaluation. Some discussion 
of these procedures and forms might be appropriate to include in a manual for 
authors of CAI materials." The procedures and forms are as follows. 

First, the Baker and Schultz cycle for instructional materials development 
provided an appropriate framework for scheduling activities for course development. 
The model- served the purposes of formative evaluation very satisfactorily. 
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Second, time spent on-line learning the capabilities of the 1500 system gave 
the author ideas about how system capabilities could be used effectively, about the 
disadvantages and advantages of certain formats for content presentation and 
queries, and about various forms of and reaction to feedback for both correct and 
incorrect responses. This is a simple idea but how many times is it overlooked? 
When an author and a software programner work in consort to develop a CAI lesson, 
the autho; should spend the maximum permissible time on-line. 

Third, author revisions done prior to submitting lesson materials to the 
CAI Laboratory eliminated changes and corrections which would otherwise have 
consumed time either on-line or in discussion with a software programmer in order 
to clarify the author's intent. 

Fourth, outside evaluation was facilitated by use of coiment cards devised 
by the author. The cards, with spaces for identifying the lesson label and medium, 
reduced the author's time needed to identify the material being evaluated and 
commented upon. The comment cards were chosen by Laboratory personnel to replace 
blank cards previously used, which was another indicator of usefulness. Comment 
card formats are given in Appendices G-2 and G-3 of the report. 

Examples of author format sheets, audio message program sheets, and a CRT 
coding sheet for development of CAI lessons are given in Appendix I of the report. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references; 



Computer-assisted instruction 
Individualized Instruction 

Formative evaluation by twelve outside evaluators 
Civilian 

3.5 Validate instruction 

Procedure 

146 

41 
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REFERENCE NUMBER: 4.1.1 



IMPLEMENT INSTRUCTIONAL MANAGEMENT PLAN 



CITATION: 



Air Force Advanced Instructional System, Weapons Mechanic implementation plan 
(Revision 4). St. Louis, Mo.: McDonnell Douglas Astronautics Company - 
East, 30 September 1975. 



An implementation plan is a detailed document covering every aspect of 
conducting and administering a course of instruction. This particular plan was 
written for a course in a computer-assisted, computer-managed, mul ti-mefiia, multi- 
track prototype instructional system being developed for the Air Force Air Training 
Command. Topics covered by the Weapons Mechanic implementation plan include: 
(a) development and validation of instructional materials; (b) manual management 
procedures; (c) development and validation of entry testing procedures; (d) sequence 
and structure of lesson materials, including alternative learning strategies; (e) 
resource requirements for instruction; (f) media overlap, multi-track, and computer- 
assisted instruction strategies; (g) incentive management procedures; (h) acceptance 
tests for instructional materials; (i) computer-managed instruction capabilities 
and application; (j) media, media devices, and individualized carrels; (k) main- 
tenance, testing, and procurement of media devices; and (1) facil ity modification 
and hardware/equipment support. 

Since the specific detail in an implementation plan cannot be widely trans- 
ferred as an authoring tool or .procedure, this discussion will center on three 
characteristics of value to other course developers: (1) the kind of detail 
required in an implementation plan; (2) the sequential treatment of instructional 
components; and (3) the support documentation accompanying the narrative text. 

First, the detail in an implementation plan consists of concrete, specific 
statements which allow personnel to implement the intended course and, more impor- 
tantly, to identify possible flaws in that implementation. For example, the 
Weapons Mechanic implementation plan specifies the exact levels of review for 
instructional materials and in what functional departments lesson reviewers work. 
In defining resource requirements for large-group tryouts, the implementation plan 
specifies, "The primary resource which shall be required for large-group tryouts is 



ABSTRACT: 



B-130 




AUTHORING AID ^^^^ 
BIBLIOGRAPHY VOLUmE I 1 1 FEBRUARY 1977 

instructors. The procedures for using instructors to collect data for dependent 
variables and criteria are described in the Section on Manual Management Pro- 
cedures. At least 2 instructors per block, shift, and week will be required. 
Their roles are described in the Section on Manual Management Procedures." And 
concerning the maintenance of carrels, the implementation plan states, "Based on 
the nature of carrels (e.g., no moving parts, no primary optical alignments, etc.) 
preventative maintenance shall be performed on a 40-day cycle. . .Preventative 
maintenance shall include inspection for surface damage, structural damage and 
rigidity. Figure 3-1 shows the logic tree for preventative maintenance." In 
other words, the clearly stated procedures and requirements in an implementation 
plan are intended to prevent the confusion and unintentional mistakes that result 
from unplanned initiation of an instructional system or its component parts. 

Second, by using a systematic, sequential treatment of component parts, very 
few details are "forgotten" or buried. For example, in this document, the section 
on carrels reads, "3.2 Carrels, 3.2.1 Implementation of Carrels, 3.2.2 Maintenance 
of Carrels, 3.2.3 Quality Control of Carrels, 3.2.4 Testing of Carrels." Other 
sections follow the same format. It may read mechanically, but the thoroughness 
and detail assured by such a format are required in an implerrentation plan. 

Third, the support documentation accompanying an implementation plan is 
designed to answer possible questions raised in the text and to provide a factual 
basis for the scheme of the plan. For example, in this document it is stated that 
"During the initial tryout period, three methods are explained in the text. One 
method is by personal interview, for which Appendix A contains a structured 
questionnaire." Similar documentation in the form of figures and tables also 
exists for course outline, small-group tryout schedule, facility modification, 
computer terminal installation, and other activities discussed in the text. ' 

An effective implementation plan will have these three characteristics of 
great detail, logical treatment of component parts, and support documentation. 
Once a complex operation 1 ike implementing an instructional system is "nailed 
down" with an implementation plan, system developers and system users have a 
common ground for reviewing and refining the system. 
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DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references; 



Computer-assisted instruction, audio-visual , and 
programmed text 
IndividuaJized instruction 
None 

Military (Air Force) 

4.1 Implement Instructional Management Plan 

Procedure 

75 

None 
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REFERENCE NUMBER: 4.2.1 CONDUCT INSTRUCTION 

CITATION: 

Air Force Advanced Instructional System. Learning center instructor (LCI) . 

St. Louis, Mo.: McDonnell Douglas Astronautics Company - East, 1 March 1976. 

ABSTRACT: 

A self-paced, individualized, instructor-training package was written for use 
in a computer-assisted, computer-managed, multi-media, multi-track prototype 
instructional system being developed for the Air Training Command. The 
package is divided into two parts. The first part consists of textual discussion 
concerning the instructor functions in the Advanced Instructional System (AIS). 
The second part consists df performance-orinented internship and certification 
training documents for three specific courses: (a) Inventory Management and 
Materiel Facilities; (b) Precision Measurement Equipment; and (c) Weapons Mechanic. 

The first part of the package contains: (a) a glossary of terms related to 
computer-based instruction; (b) an introduction to the AIS; (c) a discussion of the 
functions and responsibilities of learning center instructors; (d) details, pro- 
cedures, and input forms for the computer-managed portion of the system; (e) a 
discussion of how to maintain favorable learning conditions in the learning center; 
and (f) a discussion of programmed texts and audio-visual modules for instructional 
use in the AIS. 



, The second part of the package is a procedural i zed guide to instructor duties 
in a learning center. Topics covered inc ide: (a) computer terminal operations 
for grading and for learning center management; (b) instructional preparations and 
student-centered activities in learning center operations; (c) entry testing pro- 
cedures; (d) administrative duties required of learning center instructors; and 
(e) curriculum development responsibilities for lesson materials reviewing and 
career progression. 
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DATA SUMMARY: 

Media: Print 

Application: Individualized instruction 

Evaluation: , None 

Environment: ' Military (Air Force) 

Key number: 4.2 Conduct Instruction 

Authoring aid type: Procedure 

Number of pages: 70+ 

Number of references: None 
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REFERENCE NUMBER: 4.2.2 CONDUCT INSTRUCTION 

CITATION: 

Askren, W. B., & Valentine, R. I. Value of .iob experience to teaching effective- 
ness of technical training instructors (Report No. AFHRL-TR-70-8) . 
Wright. Patterson Air Force Base, Ohio: Air Force Human Resources Laboratory, 
Training Research Division, June 1970. (NTIS Document Reproduction Service 
No. AD 709 876) 

ABSTRACT: 

Is teaching effectiveness related to field job experience? After comparing 
Air Force technical training instructors with and without field experience, the 
authors found no evidence of significant differences in teaching effectiveness 
between the two groups of instructors. However, grades for students who had 
field-experienced instructors increased from start to finish of course, while grades 
for students who had non-field-experienced instructors decreased in the same period. 

Both groups of instructors were also rated in twelve categories by supervisors. 
Field-experienced instructors were generally rated higher than non-field-experienced 
instructors in three categories: (a) Knowledge of Subject Matter; (b) Student 
Interest; and (c) Student Participation. No difference was noted in the instructor 
ratings for four categories: (a) Lesson Plan Preparation; (b) Organization of 
Material; (c) Use of Training Aids; and (d) Evaluation of Student Progress. 

Interaction between type of instructor and course unit indicated that non-field- 
experienced instructors were more effective with lecture-theory units. Field- 
experienced instructors were more effective with equipment-oriented units. 

DATA SUMMARY: 

Media: None 

Application: Conventional instruction 

Evaluation: None 

Environment: Military (Air Force) 

Key number: 4.2 Conduct Instruction 

Authoring aid type: Procedure 

Number of pages: 

1 79 

Number of references: 2 ^ ^ 
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REFERENCE NUMBER: 4.2.3 CONDUCT INSTRUCTION 

CITATION: 

Popham, W. J., & Baker, E. L. Classroom instructional tactics . Englewood 
Cliffs, N.J.: Prentice Hall, 1973 

ABSTRACT: 

Learning and attitude are closely related. Since 'uctors wish ,to promote 
reasonable attitudes in students (and thus create a bf.i . ^-znvironment for learning), 
repressive techniques in class management should be avoided. Popham and Baker 
suggest six rules for rational classroom management based on educational and 
psychological research. 

Rule No>'J . Attend to observable behavior . Criteria for judging the 
propriety of students' actions should be established, of course, but simply 
because you "sense trouble" is seldom a reason for punishment. 

Rule No. 2 . Tell students what type of behavior is expected of them. This 
information should not be expressed in a threatening manner nor should it be - 
hidden and then released bit by bit when a student is caught misbehaving. 

Rule No. 3 . Do not reward undesirable behavior. 

Rule No. 4 . Avoid punishing undesirable behavior. 

Rule No> 5 . Provide acceptable alternative student responses for undesirable 
behavior. Rules 3, 4, and 5 are closely linked to each other. The idea behind 
them is: if you^want a behavior to continue^ reward it; if you want a behavior to 
disappear, ignore it and offer an alternative response. 

Rule No. 6 . Reward appropriate behaviors. When a studant has substituted or 
has begun to substitute a desirable response for an undesirable response, reward him. 
An appropriate reward called a positive reinforcer depends upon the context 
of the individual student and the learning situation. 
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The rules are presented in a printed, self-instructional program complete with 
an answer mask and a mastery test with answers. A coordinated filmstrip-tape 
program is also available from Vincet Associates, Inc., P.O. Box 24714, Los Angeles 
California 90024. 

DATA SUMMARY: 

Media: Programmed text (audio-visual is optional) 

Application: Conventional and individualized instruction 

Evaluation: None 

Environment: Civilian 

Key number: 4.2 Conduct Instruction 

Authoring aid type: Tool 

Number of pages: 158 

Number of references: None 



181 

B-137 



AfOOO/VA/fLf. DOUGLAS ASmOMAtrTICS COMfAIVV' EAST 



AUTHORING AID 
BIBLIOGRAPHY 



VOLUME I 



MDCE1570 
11 FEBRUARY 1977 



REFERENCE NUMBER: 5.1.1 



CONDUCT INTERNAL EVALUATION 



CITATION: Bailey, R. W. Testing manual procedures in computer-based business 



Proceedings of the 16th Annual Meeting of the Human Factors Society , 1972, 
395-401 

ABSTRACT: . 

Testing of .nanual procedures as an essential part of the total systems 
development process is discussed. The Bell System personnel subsystem testing 
program is used as an example. It consists of six steps. The first step is 
Preliminary Evaluation, which Gonstists of design verification of personnel sub- 
system products by checklists and low-level simulation techniques. 

The second step is a Position Package Test which evaluates position packages 
by using representative subjects in a simulated operational environment. A 
position package consists of position practice, training materials, performance 
aids, forms and/or information displays, and a statement of the minimum qualifica- 
tions required of position holders. Position may be defined as job responsibility. 

The third step is a Subsystem Test which analyzes the proper information flow 
for processing every transaction that enters a system and triggers system processing 
Sequential information flow into and out of positions is also considered. 

The fourth step is a System Test, consisting of performance tests for the 
personnel subsystem and the computer subsystem in separate simulated operational 
environments. 

The fifth step is a Total System Test. The tPtal business information system 
is performance-tested in a simulated operational environment. 



information systems. In W. B. Knowles, M. S. Sanders, & F. A. Muckler (Eds.), 
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The sixth step is a Trial. The final stop of the 'testing program is conducted 
in a real-time- operational environment to determine if the system achieves specified 
system performance objectives under normal operating conditions. 

A test report containing quantitative results and recomriendations records not 
only the number and kinds of errors made during the test but also the probable 
causes for the errors. Accuracy objectives are set before testing begins, and 
estimated error rates derived from the tests are compared with the accuracy 
objectives to determine what changes are required to make the system meet its 
performance objectives. 

author notes that errors can be caused by seven factors: (a) design j (b) 
doc i» ation; (c) training; (d) personal ; (e) source data; (f) environmental; 
and (g) m€in-/?iachipe interface. 

DATA SUIW^^Y: 

Media: Simulation 

Application: Conventional and individualized instruction 

Eviludtion: None 

Environment: Civilian 

Key number: 5.1 Conduct Internal Evaluation • 

Aii'vhoring aid type: PT>cf::iure 

NuiT^ber of pages: 7 

Number' t i factors- i^ce 
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REFERENCE NUMB:'^: 5.1 : 



CONDUCT INTERNAL EVALUATION 



CITATION: 



Miller, G. G., and Sellman W. S. Development of psychometric measures of student 
attitudes toward technical training: Norm group report (Report No. AFHRL 
TR-73-15). Lowry Air Force Base, Colorado: Air Force Human Resources 
Laboratory, Technical Training Division, October 1973. 



Development of a new student critique form (SCF) for the Air Training Command 
(ATC) is discussed. The study had four objectives: (a) generate more items for 
a previously-developed prototype scale in order to increase its reliability; 
(b) administer the improved SCF to large numbers of officer, NrOs and airmen 
enrolled in technical courses; (d) factor analyze the SCF in order to ensure scale 
unidimensional ity; and (d) norm the SCF for officers, NCOs, and airmen. Factor 
analyses revealed seven unidimensional scales: (a) Instructor Competence; 
(b) Study Environment and Testing; (c) Specialty Training; (d) Training Impressions; 
(e) Classroom Facilities and Environment; (f) Training Devices and Audio-Visual 
Aids; and (g) Training Materials Adequacy. 

A 69- item SCF was administered to officers, NCOs, and airmen enrolled in 
technical training courses at six Air Force bases (Chanute, Keesler, Sheppard,' 
Lowry, Lackland, and Goodfellow). Scores were returned for each rank-. Air Force 
Specialty Code (AFSC), and Air Force base where the training center was located, 
as individual average scores and as scale scores. Scores for each unidimensional 
scale were also reported by rank, AI'.^C, and base. 

Analyses of scores returned by 1,669 trainees indicated: (1) attitudes of 
trainees toward technical training were "slightly" positive; (2) the most 
positive attitudes toward tachnical training were held by NCOs. followed by 
officers, and then airmen; (3) since statistically significant differences existed 
among scores at' the six bases, the SCF should be normed separately for each base; 
(4) the Instructor Competence scale received the most consistently-positive 
response from trainees; and (5) the Specialty Training Scale received the most 
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consistently-negative response from trainees, which indicated a lack of assurance — 
especially among airmen — that technical training was adequately preparing them 
for on-the-job performance. 

The next phase of the critique improvement program will be development and 
implementation of computer software for managing the enti re student critique 
program. 

Appendix A contains a sample ATC Form 736, Student Critique. Appendix El 
contains the improved student critique form. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
NuP't :r of references: 



Print 

Conventional and individualized instruction 

Internal statistical analysis 

(^iilitary (Air Force) 

'^.1 Cor duct Internal Evaluation 

Tool 



ERIC 



B-141 

185 



AUTHORING AID 
BIBLIOGRAPHY 



VOLUME I 



MDC El 570 
11 FEBRUARY 1977 



REFERENCE NUMBER: 5.2.1 



CONDUCT EXTERNAL EVALUATION 



CITATION: 



Anastasio, E.J. An evaluation of the demonstrations being conducted by The 

University of Illinois and the MITRE Corporation of their respective computer 
assisted instructional systems (ETS PR-72-19). Princeton, M.J.: Educational 
Testing Service, 1972. (ERIC Document Reproduction Service No. ED 072 070) 



Development and structure of a plan for evaluating the cost performance and 
educational benefits of the TICCIT and PLATO projects in 1971-72 are reported. 
Included with the report are: (a) an annotated bibliography of documents relating 
to computer-based education in general, and to research and development and 
evaulation concerns of each project; (b) Appendix A, Survey of Instructor 
Activities and Attitudes; and (c) Appendix B, Attitudes toward Reading Question- 
naire — Grade 2 and Grades 4 and 6. Due uo schedule slippages in each project's 
development, the field-test evaluation has been repeatedly postponed. However, as 
an example of evaluation procedure, -^the Educational Testing Service (ETS) report is 
useful. The report is divided into three sections. 

The first section discusses cost analysis of PLATO and TICCIT in terms of five 
cost categories: (a; :idrdware; (b) software; (c) courseware; (d) administration; 
and (e) incremental site (out-of-pocket). A distinction is made between one-time 
costs (developmental and implementation expenses) and recurring costs (operating 
costs). In the TICCIT cost analysis section, there are four parts to hardware 
costs: (a) the processor facility; (b) the communication system; (c) the terminal; 
and (d) audio-video and other, including personnel costs. There are three parts 
to software coses: (a) application software; (b) authoring software; and (c) 
delivery software. Courseware costs are figured in dollars-per-hour for course- 
ware development by the following specialists: (a) instructional psycholcgi^> • 
(b) instructional design technician; (c) empirical design technician; [6) pa^-kuging 
specialist; (e) author; and (f) program implementation. Administration costs are 
determined monthly on the basis of the ratio of each course-development cost of 
the total monthly developmental costs. Incremental costs include expense^s for modi- 
fying a site for instruction (including air conditioning, and electrical and 
facility modification). Cost categories for the PLATO system have been similarly 
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divided, although it is noted that the very different natures of the two systems 
make direct comparisons misleading. 

The second section discusses educational analysis of the two systems in" terms 
of: (a) achievement and attitude measurement; (b) experimental design; and (c) 
data collection. Student achievement would be measured by standardized achievement 
tests, with addition of item-analysis and multiple-matrix sampling techniques for 
obtaining group measurement. Student attitude would be measured by five standard- 
ized attitude surveys and by ETS-developed items. The five attitude surveys are: 
(a) Student Instructional Report; (b) Comparative Guidance and Placement Program 
Student Questionnaire; (c) College Student Questionnaire-Part 2; (d) College and 
University Environment Scales; and (e) Student Reactions to College. Several of 
these scales were still under development in 1972. Not all scales would be completed 
by any one student. Faculty attitudes would also be assessed. Experimental design 
considers the comparability of groups taking different treatments, comparability of 
content taken by taking different treatments, obtaining matched pairs of students 
with and without CAI, and appropriate statistical analyses to assess CAI program 
effectiveness. Data would be collected from each project's own records concerning 
-vstem cost and maintenance, i.e., system reliability. In addition, system per- 
formance variables such as terminal response times, queueing effects, ease of up- 
dating both software and hardware, and user satisfaction with the system would be 
examined. The terminals would be examined for comfort, safety, durability, and 
special environmental or wiring requirements. 

The third section covers technical analysis, which consisted of a number of 
visits by '-TS to TICCIT and PLATO for detailed briefings on the projected hardware 
and system design. No conclusive findings are given. 

DATA SUMMARY: 

Media: Computer assisted -i nstruction, computer-managed 

instruction 

Application: Self-paced, individualized instruction 

Evaluation: None 
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Number of references: 107 
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REFERENCE NUMBER: 5.3.1 REVISE SYSTEM 

CITATION: 

Havelock, R G., Havelock, M.C., & Markowitz, E.A. Educational innovation in the 

United States. Volume I: The national survey: The substance and the process . 

Ann Arbor: The University of Michigan, Institute for SociaV Research, Center 
for Research on Utilization of Scientific Knowledge, June 1973. 

ABSTRACT: 

This report is an assessment of educational innovations in the public sector 
during the 1970-71 school year. The process by which those innovations come about 
or continue to operate is discussed. 

Specific findings, which may be of some interest, are as follows: (a) of 353 
school districts sampled, 346 (98%) reported at least or-^* major innovation during 
the year, with "major" being defined as a "substantial reorientation on the part of 
the staff, a reallocation of resources, or adoption of new practices, programs, or 
technology," (b) individualized instruction and team teaching were the most 
frequently reported innovations; (c) 83% of the innovations were successful; (d) 
teachers played more key roles in the innovations, followed by assistant super- 
intendents, and then principals; (e) the greater barrier to innovations was 
co:;fusion among staff about the purpose of the innovation; and (f) school district 
size and per-pupil expenditure positively correlated with frequency of innovation, 
but no relationship was found between innovativeness and the percent of graduates 
going on to four year colleges, the pupil-teacher ratio, or the difficulty in 
obtaining financing for new or existing programs. 

The data for this study were obtained through a very detailed but well -formatted 
mail questionnaire, which is given in Appendix A with an instruction sheet. The 
questionnaire could be easily adapted for obtaining data for revising an 
instructional system. * The three-page questionnaire is designed to gather data: 
(a) describing the nature of the innovation and how it came about; (b) how much 
emphasis was given to several key innovation procedures; (c) what barriers to 
innovations existed; (d) what institutional changes in both the local school and school 
district resulted from the innovation; (e) what internal and external resources 
were used to implement the innovation; (f) how often and to what extent the 
innovation was used; and (g) specific characteristics of the school district, 
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Including information on policies and procedures, sysLoin size and staffing, and 
frequency of teacher strikes and student-community group protests. 



DATA SUMMARY: 
Media: 

Application: 
Evaluation: 
Environment: 
Key number: 
Authoring aid type: 
Number of pages: 
Number of references; 



Print 

Conventional and individualized instruction 
None 

ivilian • 
5.3 Revise system 
Tool 

234+ (Report), 3 (Questionnaire) 
24 (Report) 
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REFERENCE NUMBER: 5.3.2 REVISE SYSTEM 

CITATION: 

Haverland, E. M. Transfer and use of training technology: A model for matching 
training a pproaches with training settings (HumRRO Tech. Rep. 74-24). 
Alexandria, Va.: Human Resources Research Organization, October 1974. 
(NTIS Documer' Reproduction Service No. AD A005 816) 

ABSTRACT: 

A model for evaluating training approaches or innovations in terms of 
requirements, resources, and constraints of specific training settings was 
developed and field-tested. The model contains two parallel sets of questions. 
The first set of questions concerns the characteristics of the training approach 
under consideration. The second set of questions concerns the requirements, 
resources, and constraints in the specific training setting . This report 
discusses the development of the preliminary version of the model, its field 
evaluation by Air Force training personnel, and its subsequent revision. A 
stand-alone version of the model with supporting material is given in the text. 
Reationship of the model to Air Force procedures for Instructional System 
Development (ISD), to the Rand Corporation's design system for instructional sys- 
tems (MODIA), and to the Air Force's Advanced Instructional System (AIS) is 
discussed. The application of the model to several different training approaches 
and setting is discussed. 

With matching, parallel sets of questions, the model operationally covers the 
following components of instructional systems: (a) resources required and 
resources available; (b) instructional design and management; (c) training content 
characteristics; (d) instructional personnel; and (e) student characteristics. 

The model was applied to the training approach called peer instruction used 
in the Law Enforcement Specialist course (3ABR81230) at Lackland Air Force Base- 
Information for the model came from the Course Chart and the Plan of Instruction 
for the course. General information was gathered during a site visit to the 
School of Applied Aerospace Sciences at Lackland Air Force Base. 
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Because required information cojIcJ m;. t be obtained from this course, the 
application of the model was unsatisfactory > but plans were made to apply the 
model to another training setting in close collaboration with Air Force personnel. 
Critical data which could not be gathered during the original tryout included: 
(a) the objectives of persons working in the training setting who were responsible 
for its efficient and effective operation; (b) problems existing within the train- 
ing setting; (c) availability of space, equipment, and support services; (d) lead 
times for course modifications; (e) school operating policies; (f) student char- 
acteristics; (g) availability of funds in the training se:^-ing; and (h) character- 
istics of the training content. It was concluded that io^^ \m could not be 
applied very successfully to a training setting at a di and with documentation 

intended for other uses . In the proposed on-site app.ica«Mon of the model, a more 
rigorous evaluation of the model will be conducted. 

Results of the original appl ication of the model ^o the peer instruction 
training approach are given in Appendix A. 



DATA SUMMARY: 

Media: Print 

Application: Peer instruction 

Evaluation: Field test 

Environment: Military (Air Force) 

Key number: 5.3 Revise system 

Authoring aid type: o..,Xopl 

Number of pages: 7&' 
Number of references: 24 
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